Jun 01 11:23:03.250847 np0000001841 systemd[1]: Started devstack@neutron-ovn-maintenance-worker.service - Devstack devstack@neutron-ovn-maintenance-worker.service. Jun 01 11:23:04.740150 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.common.config [-] Logging enabled! Jun 01 11:23:04.740525 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.common.config [-] /opt/stack/data/venv/bin/neutron-ovn-maintenance-worker version 28.1.0.dev308 Jun 01 11:23:04.740525 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.common.config [-] command line: /opt/stack/data/venv/bin/neutron-ovn-maintenance-worker --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini {{(pid=88598) setup_logging /opt/stack/neutron/neutron/common/config.py:125}} Jun 01 11:23:04.741411 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.server.ovn_maintenance [-] OVN maintenance process starting... Jun 01 11:23:04.741558 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "plugin-directory" by "neutron_lib.plugins.directory._create_plugin_directory" {{(pid=88598) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:511}} Jun 01 11:23:04.741669 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG oslo_concurrency.lockutils [-] Lock "plugin-directory" acquired by "neutron_lib.plugins.directory._create_plugin_directory" :: waited 0.000s {{(pid=88598) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:524}} Jun 01 11:23:04.741791 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG oslo_concurrency.lockutils [-] Lock "plugin-directory" "released" by "neutron_lib.plugins.directory._create_plugin_directory" :: held 0.000s {{(pid=88598) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:543}} Jun 01 11:23:04.741883 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG oslo_concurrency.lockutils [-] Acquiring lock "manager" by "neutron.manager.NeutronManager._create_instance" {{(pid=88598) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:511}} Jun 01 11:23:04.741954 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG oslo_concurrency.lockutils [-] Lock "manager" acquired by "neutron.manager.NeutronManager._create_instance" :: waited 0.000s {{(pid=88598) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:524}} Jun 01 11:23:04.742082 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.manager [-] Loading core plugin: ml2 Jun 01 11:23:05.167460 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.167665 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.167665 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.168312 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.168414 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.168504 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.208416 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.208516 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.208665 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.218213 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.218213 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.218213 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.267075 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.267130 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.267250 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.273691 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.273818 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.273818 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.340145 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.340286 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent after_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.340286 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: segment precommit_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.340404 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: network precommit_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.340657 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: segment before_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.397450 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network before_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.397579 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: agent before_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.397667 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network before_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.397745 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.397789 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.397874 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment precommit_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.397939 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment precommit_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.398012 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.398077 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.398116 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.398202 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > network precommit_delete 0, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.398272 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port provisioning_complete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.398347 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.398388 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.398460 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > agent after_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.398520 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.398594 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet precommit_delete 0, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.398668 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.398716 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.398795 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.398874 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > agent after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.398968 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.399038 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.399114 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > rbac-policy before_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.399432 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: network configured {{(pid=88598) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 01 11:23:05.399481 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: port configured {{(pid=88598) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 01 11:23:05.399578 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: subnet configured {{(pid=88598) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 01 11:23:05.399642 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: subnetpool configured {{(pid=88598) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 01 11:23:05.399706 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: security_group configured {{(pid=88598) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 01 11:23:05.399706 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.quota.resource_registry [-] Tracking information for resource: security_group_rule configured {{(pid=88598) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 01 11:23:05.399975 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [-] Configured type driver names: ['local', 'flat', 'vlan', 'geneve'] Jun 01 11:23:05.400905 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.drivers.type_flat [-] Allowable flat physical_network names: ['public'] Jun 01 11:23:05.401619 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.drivers.type_local [-] ML2 LocalTypeDriver initialization complete Jun 01 11:23:05.402428 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.drivers.type_vlan [-] Network VLAN ranges: OrderedDict({'public': [(1, 4094)]}) Jun 01 11:23:05.402509 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [-] Loaded type driver names: ['flat', 'geneve', 'local', 'vlan'] Jun 01 11:23:05.402694 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [-] Registered types: dict_keys(['flat', 'geneve', 'local', 'vlan']) Jun 01 11:23:05.402788 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [-] Project network_types: ['geneve'] Jun 01 11:23:05.402896 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [-] Configured extension driver names: ['port_security', 'qos'] Jun 01 11:23:05.403471 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [-] Loaded extension driver names: ['port_security', 'qos'] Jun 01 11:23:05.403526 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [-] Registered extension drivers: ['port_security', 'qos'] Jun 01 11:23:05.403646 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [-] Configured mechanism driver names: ['ovn'] Jun 01 11:23:05.447901 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [-] Loaded mechanism driver names: ['ovn'] Jun 01 11:23:05.448023 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [-] Registered mechanism drivers: ['ovn'] Jun 01 11:23:05.448071 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [-] No mechanism drivers provide segment reachability information for agent scheduling. Jun 01 11:23:05.448783 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_response 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.448867 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port before_response 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:05.472880 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'flat' Jun 01 11:23:05.473001 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.drivers.type_flat [-] ML2 FlatTypeDriver initialization complete Jun 01 11:23:05.473001 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [-] Initializing driver for type 'geneve' Jun 01 11:23:05.473112 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.drivers.type_tunnel [-] geneve ID ranges: [(1, 65536)] Jun 01 11:23:05.513263 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO dbcounter [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Registered counter for database neutron Jun 01 11:23:05.557511 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG oslo_db.sqlalchemy.engines [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=88598) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.12/site-packages/oslo_db/sqlalchemy/engines.py:325}} Jun 01 11:23:05.962580 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG dbcounter [-] [88598] Writer thread running {{(pid=88598) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:102}} Jun 01 11:23:06.096056 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Initializing driver for type 'local' Jun 01 11:23:06.096138 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Initializing driver for type 'vlan' Jun 01 11:23:06.192395 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.drivers.type_vlan [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] VlanTypeDriver initialization complete Jun 01 11:23:06.192395 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Initializing extension driver 'port_security' Jun 01 11:23:06.192395 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.extensions.port_security [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] PortSecurityExtensionDriver initialization complete Jun 01 11:23:06.192395 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Initializing extension driver 'qos' Jun 01 11:23:06.192395 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.plugins.ml2.extensions.qos [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] QosExtensionDriver initialization complete {{(pid=88598) initialize /opt/stack/neutron/neutron/plugins/ml2/extensions/qos.py:31}} Jun 01 11:23:06.192395 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Initializing mechanism driver 'ovn' Jun 01 11:23:06.192395 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Starting OVNMechanismDriver Jun 01 11:23:06.193921 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > process before_spawn 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.193921 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > process after_init 55550000, True {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.193921 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > segment after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.194012 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > segment after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.194067 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > segment after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.194130 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > allowed_address_pair before_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.194224 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > security_group precommit_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.194304 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > security_group after_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.194379 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > security_group after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.194460 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > security_group precommit_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.194586 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > security_group after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.194675 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > security_group_rule precommit_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.194753 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > security_group_rule after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.194830 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > security_group_rule before_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.194905 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > address_group after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.194981 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > address_group after_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.195055 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > address_group after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.195217 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > qos_plugin after_init 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.195380 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > trunk_plugin after_init 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.195565 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > logging-plugin after_init 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.195936 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.services.logapi.drivers.ovn.driver [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] OVN logging driver registered Jun 01 11:23:06.196021 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: ._register_pvlan_driver at 0x7d9a4967cf40> pvlan_plugin after_init 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.197174 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.plugins.ml2.plugin [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] neutron.plugins.ml2.plugin.Ml2Plugin method _start_rpc_notifiers called with arguments () {} {{(pid=88598) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} Jun 01 11:23:06.197397 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > port after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.197489 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > port after_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.197593 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > port after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.197674 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > subnet after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.197753 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > subnet after_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.197826 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > subnet after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.197901 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > network after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.197978 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > network after_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.198051 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > network after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.198125 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > security_group after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.198205 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > security_group after_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.198276 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > security_group after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.198348 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > security_group_rule after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.198419 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > security_group_rule after_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.198490 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > security_group_rule after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.198581 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > address_group after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.198669 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > address_group after_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.198729 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > address_group after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.198803 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.plugins.ml2.ovo_rpc [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ML2 OVO RPC backend initialized. {{(pid=88598) __init__ /opt/stack/neutron/neutron/plugins/ml2/ovo_rpc.py:117}} Jun 01 11:23:06.199511 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.quota [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded quota_driver: . Jun 01 11:23:06.199626 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.plugin [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Modular L2 Plugin initialization complete Jun 01 11:23:06.199733 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading services supported by the core plugin {{(pid=88598) _load_services_from_core_plugin /opt/stack/neutron/neutron/manager.py:170}} Jun 01 11:23:06.199820 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.plugins.ml2.managers [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Got port-security extension from driver 'port_security' Jun 01 11:23:06.200024 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading service plugins: ['ovn-router', 'auto_allocate', 'tag', 'timestamp', 'network_ip_availability', 'flavors', 'revisions'] {{(pid=88598) _load_service_plugins /opt/stack/neutron/neutron/manager.py:195}} Jun 01 11:23:06.200095 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading Plugin: ovn-router Jun 01 11:23:06.207631 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > router before_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.207740 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: port before_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.207817 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: port after_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.207910 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > port precommit_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.207988 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: port before_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.208062 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: port before_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.208140 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > router precommit_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.208228 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > router precommit_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.208306 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > router precommit_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.208390 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: subnet after_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.208559 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.quota.resource_registry [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Tracking information for resource: router configured {{(pid=88598) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 01 11:23:06.208640 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.quota.resource_registry [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Tracking information for resource: floatingip configured {{(pid=88598) set_tracked_resource /opt/stack/neutron/neutron/quota/resource_registry.py:203}} Jun 01 11:23:06.208748 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.services.ovn_l3.plugin [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Starting OVNL3RouterPlugin Jun 01 11:23:06.208904 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > port_forwarding_plugin after_init 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.209060 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > router before_create 55550100, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.209143 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > router precommit_delete 55550100, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.209233 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > router precommit_create 55550100, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.209311 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > router precommit_update 55550100, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.211431 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.services.provider_configuration [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Service providers = [] {{(pid=88598) parse_service_provider_opt /opt/stack/neutron/neutron/services/provider_configuration.py:170}} Jun 01 11:23:06.211517 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.db.servicetype_db [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Adding provider configuration for service L3_ROUTER_NAT {{(pid=88598) add_provider_configuration /opt/stack/neutron/neutron/db/servicetype_db.py:45}} Jun 01 11:23:06.213102 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > router precommit_create 55549900, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.213209 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > router_interface after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.213296 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > floatingip after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.213373 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > floatingip precommit_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.213449 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > floatingip precommit_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.213523 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > floatingip after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.213633 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > floatingip after_status_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.213707 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > floatingip after_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.213787 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > port_forwarding after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.213856 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > router_interface after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.213934 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > router after_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.214009 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > router precommit_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.214082 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > router after_delete 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.214158 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > router after_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.214274 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.services.service_base [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded 'neutron.services.ovn_l3.service_providers.ovn.OvnDriver' provider for service L3_ROUTER_NAT {{(pid=88598) load_drivers /opt/stack/neutron/neutron/services/service_base.py:49}} Jun 01 11:23:06.214364 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: > process after_init 55550001, True {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.214482 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Successfully loaded L3_ROUTER_NAT plugin. Description: L3 Router Service Plugin for basic L3 forwarding using OVN {{(pid=88598) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 01 11:23:06.214557 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading Plugin: auto_allocate Jun 01 11:23:06.215372 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: network precommit_update 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.215465 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron_lib.callbacks.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Subscribe: network precommit_create 55550000, False {{(pid=88598) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.216038 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Successfully loaded auto-allocated-topology plugin. Description: Auto Allocated Topology - aka get me a network. {{(pid=88598) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 01 11:23:06.216038 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading Plugin: tag Jun 01 11:23:06.217972 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Successfully loaded TAG plugin. Description: Tag support {{(pid=88598) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 01 11:23:06.218052 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading Plugin: timestamp Jun 01 11:23:06.219077 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Successfully loaded timestamp plugin. Description: Adds timestamps to Neutron resources with standard attributes {{(pid=88598) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 01 11:23:06.219152 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading Plugin: network_ip_availability Jun 01 11:23:06.221932 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Successfully loaded network-ip-availability plugin. Description: Provides IP availability data for each network and subnet. {{(pid=88598) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 01 11:23:06.221932 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading Plugin: flavors Jun 01 11:23:06.221932 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Successfully loaded FLAVORS plugin. Description: Neutron Flavors and Service Profiles manager plugin {{(pid=88598) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 01 11:23:06.221932 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading Plugin: revisions Jun 01 11:23:06.222213 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.manager [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Successfully loaded revision_plugin plugin. Description: Adds revision numbers to resources. {{(pid=88598) _create_and_add_service_plugin /opt/stack/neutron/neutron/manager.py:239}} Jun 01 11:23:06.222321 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG oslo_concurrency.lockutils [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Lock "manager" "released" by "neutron.manager.NeutronManager._create_instance" :: held 1.480s {{(pid=88598) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:543}} Jun 01 11:23:06.222530 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] get_extension_paths = OrderedDict({'/opt/stack/neutron/neutron/extensions': 1}) {{(pid=88598) get_extensions_path /opt/stack/neutron/neutron/api/extensions.py:640}} Jun 01 11:23:06.222631 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Initializing extension manager. Jun 01 11:23:06.222852 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: __init__.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.222923 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: __pycache__ {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.222995 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: aap_reject_multicast.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.223229 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Allowed address pairs reject multicast extension" alias="aap-reject-multicast" description="Reject multicast addresses in Allowed Address Pairs" updated="2026-04-01T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.223427 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension aap-reject-multicast not supported by any of loaded plugins Jun 01 11:23:06.223502 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: address_group.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.223821 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Address group" alias="address-group" description="Support address group" updated="2020-07-15T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.223962 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: address-group Jun 01 11:23:06.224034 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: address_scope.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.224283 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Address scope" alias="address-scope" description="Address scopes extension." updated="2015-07-26T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.224430 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: address-scope Jun 01 11:23:06.224492 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: admin_state_down_before_update.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.224705 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Enforce Router's Admin State Down Before Update Extension" alias="router-admin-state-down-before-update" description="Ensure that the admin state of a router is DOWN (admin_state_up=False) before updating the distributed attribute" updated="2019-07-02T15:56:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.224832 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension router-admin-state-down-before-update not supported by any of loaded plugins Jun 01 11:23:06.224921 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: agent.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.225163 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="agent" alias="agent" description="The agent management extension." updated="2013-02-03T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.225298 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: agent Jun 01 11:23:06.225360 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: agent_resources_synced.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.225601 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Agent's Resource View Synced to Placement" alias="agent-resources-synced" description="Stores success/failure of last sync to Placement" updated="2018-12-19T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.225720 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension agent-resources-synced not supported by any of loaded plugins Jun 01 11:23:06.225797 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: agent_sort_key.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.226050 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Agents Sort Key" alias="agent_sort_key" description="Enabling the keyword sort_key for sorting in agents." updated="2013-02-03T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.226152 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension agent_sort_key not supported by any of loaded plugins Jun 01 11:23:06.226263 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: allowedaddresspairs.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.226846 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Allowed Address Pairs" alias="allowed-address-pairs" description="Provides allowed address pairs" updated="2013-07-23T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.226973 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: allowed-address-pairs Jun 01 11:23:06.227042 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: auto_allocated_topology.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.227257 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Auto Allocated Topology Services" alias="auto-allocated-topology" description="Auto Allocated Topology Services." updated="2016-01-01T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.227375 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: auto-allocated-topology Jun 01 11:23:06.227452 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: availability_zone.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.227706 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Availability Zone" alias="availability_zone" description="The availability zone extension." updated="2015-01-01T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.227825 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: availability_zone Jun 01 11:23:06.227892 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: availability_zone_filter.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.228084 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Availability Zone Filter Extension" alias="availability_zone_filter" description="Add filter parameters to AvailabilityZone resource" updated="2018-06-22T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.228205 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension availability_zone_filter not supported by any of loaded plugins Jun 01 11:23:06.228276 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: data_plane_status.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.228468 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Port data plane status extension" alias="data-plane-status" description="Expose status of underlying data plane" updated="2017-01-24T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.228607 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension data-plane-status not supported by any of loaded plugins Jun 01 11:23:06.228691 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: default_subnetpools.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.228883 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Default Subnetpools" alias="default-subnetpools" description="Provides ability to mark and use a subnetpool as the default." updated="2016-02-18T18:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.229033 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: default-subnetpools Jun 01 11:23:06.229131 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: dhcpagentscheduler.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.229514 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="DHCP Agent Scheduler" alias="dhcp_agent_scheduler" description="Schedule networks among dhcp agents" updated="2013-02-07T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.229681 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: dhcp_agent_scheduler Jun 01 11:23:06.229752 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: dns.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.229983 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="DNS Integration" alias="dns-integration" description="Provides integration with DNS." updated="2015-08-15T18:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.230126 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: dns-integration Jun 01 11:23:06.230208 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: dns_domain_ports.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.230435 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="dns_domain for ports" alias="dns-domain-ports" description="Allows the DNS domain to be specified for a network port." updated="2017-04-24T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.230614 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: dns-domain-ports Jun 01 11:23:06.230669 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: dns_integration_domain_keywords.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.230891 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="DNS domain names with keywords allowed" alias="dns-integration-domain-keywords" description="Allows to use keywords like , , and as DNS domain name" updated="2020-06-15T18:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.231008 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: dns-integration-domain-keywords Jun 01 11:23:06.231112 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: dvr.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.231355 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Distributed Virtual Router" alias="dvr" description="Enables configuration of Distributed Virtual Routers." updated="2014-06-1T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.231489 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension dvr not supported by any of loaded plugins Jun 01 11:23:06.231593 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: ecmp_routes.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.231805 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Equal Cost Multipath Routing" alias="ecmp_routes" description="Allows traffic to reach thesame destination via multiple different links " updated="2020-11-26T18:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.231953 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension ecmp_routes not supported by any of loaded plugins Jun 01 11:23:06.232033 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: empty_string_filtering.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.232262 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Empty String Filtering Extension" alias="empty-string-filtering" description="Allow filtering by attributes with empty string value" updated="2018-05-01T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.232403 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: empty-string-filtering Jun 01 11:23:06.232482 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: evpn.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.232726 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="EVPN" alias="evpn" description="Router extension to support EVPN Type-5 route advertisement" updated="2026-04-28T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.232869 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension evpn not supported by any of loaded plugins Jun 01 11:23:06.232965 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: expose_l3_conntrack_helper.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.233191 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Expose CT target rules for conntrack helper" alias="expose-l3-conntrack-helper" description="Expose allow adding CT target rules for conntrack helper" updated="2019-04-04T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.233333 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension expose-l3-conntrack-helper not supported by any of loaded plugins Jun 01 11:23:06.233450 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: expose_port_forwarding_in_fip.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.233679 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Expose Floating IP port forwardings" alias="expose-port-forwarding-in-fip" description="Expose port forwarding fields in floating IP response" updated="2018-06-28T16:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.233804 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension expose-port-forwarding-in-fip not supported by any of loaded plugins Jun 01 11:23:06.233874 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: external_net.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.234054 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Neutron external network" alias="external-net" description="Adds external network attribute to network resource." updated="2013-01-14T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.234172 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: external-net Jun 01 11:23:06.234252 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: extra_dhcp_opt.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.234447 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Neutron Extra DHCP options" alias="extra_dhcp_opt" description="Extra options configuration for DHCP. For example PXE boot options to DHCP clients can be specified (e.g. tftp-server, server-ip-address, bootfile-name)" updated="2013-03-17T12:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.234587 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: extra_dhcp_opt Jun 01 11:23:06.234656 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: extraroute.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.234855 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Neutron Extra Route" alias="extraroute" description="Extra routes configuration for L3 router" updated="2013-02-01T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.234971 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: extraroute Jun 01 11:23:06.235036 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: extraroute_atomic.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.235246 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Atomically add/remove extra routes" alias="extraroute-atomic" description="Edit extra routes of a router on server side by atomically adding/removing extra routes" updated="2019-07-10T00:00:00+00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.235362 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: extraroute-atomic Jun 01 11:23:06.235445 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: filter_validation.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.235688 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Filter parameters validation" alias="filter-validation" description="Provides validation on filter parameters." updated="2018-07-04T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.235847 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: filter-validation Jun 01 11:23:06.235943 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: fip_pf_description.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.236145 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Floating IP Port Forwarding new attribute description" alias="floating-ip-port-forwarding-description" description="Add a description field to Port Forwarding rules" updated="2019-11-01T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.236467 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension floating-ip-port-forwarding-description not supported by any of loaded plugins Jun 01 11:23:06.236603 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: fip_pf_detail.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.236820 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Floating IP port forwarding detail" alias="floating-ip-port-forwarding-detail" description="Allow list floating ip return more port forwarding data, include'id' and 'internal_port_id'" updated="2022-05-06T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.236970 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: floating-ip-port-forwarding-detail Jun 01 11:23:06.237053 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: fip_pf_port_range.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.237279 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Floating IP Port Forwarding support port ranges" alias="floating-ip-port-forwarding-port-ranges" description="Add support to port ranges" updated="2020-07-01T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.237426 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension floating-ip-port-forwarding-port-ranges not supported by any of loaded plugins Jun 01 11:23:06.237520 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: fip_port_details.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.237778 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Floating IP Port Details Extension" alias="fip-port-details" description="Add port_details attribute to Floating IP resource" updated="2018-04-09T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.237917 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: fip-port-details Jun 01 11:23:06.238001 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: flavors.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.238255 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Neutron Service Flavors" alias="flavors" description="Flavor specification for Neutron advanced services." updated="2015-09-17T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.238395 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: flavors Jun 01 11:23:06.238475 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: floating_ip_port_forwarding.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.238780 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Floating IP Port Forwarding" alias="floating-ip-port-forwarding" description="Allow port forwarding from floating IP port to an internal port" updated="2018-01-19T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.238923 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension floating-ip-port-forwarding not supported by any of loaded plugins Jun 01 11:23:06.239015 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: floatingip_pools.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.239261 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Floating IP Pools Extension" alias="floatingip-pools" description="Provides a floating IP pools API." updated="2018-03-21T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.239411 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: floatingip-pools Jun 01 11:23:06.239493 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: ip_allocation.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.239727 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="IP Allocation" alias="ip_allocation" description="IP allocation extension." updated="2016-06-10T23:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.239869 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension ip_allocation not supported by any of loaded plugins Jun 01 11:23:06.239957 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: ip_substring_port_filtering.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.240186 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="IP address substring filtering" alias="ip-substring-filtering" description="Provides IP address substring filtering when listing ports" updated="2017-11-28T09:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.240296 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension ip-substring-filtering not supported by any of loaded plugins Jun 01 11:23:06.240362 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: l2_adjacency.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.240555 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="L2 Adjacency" alias="l2_adjacency" description="Display L2 Adjacency for Neutron Networks." updated="2016-04-12T16:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.240678 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension l2_adjacency not supported by any of loaded plugins Jun 01 11:23:06.240742 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: l3.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.241108 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Neutron L3 Router" alias="router" description="Router abstraction for basic L3 forwarding between L2 Neutron networks and access to external networks via a NAT gateway." updated="2012-07-20T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.241285 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: router Jun 01 11:23:06.241368 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: l3_conntrack_helper.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.241934 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="L3 Conntrack helper" alias="l3-conntrack-helper" description="Allow adding CT target rules for conntrack helper" updated="2019-04-04T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.242090 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension l3-conntrack-helper not supported by any of loaded plugins Jun 01 11:23:06.242189 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: l3_enable_default_route_bfd.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.242415 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Enable Default Route BFD" alias="enable-default-route-bfd" description="Enables configurable BFD behavior for default routes" updated="2023-02-27T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.242582 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: enable-default-route-bfd Jun 01 11:23:06.242669 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: l3_enable_default_route_ecmp.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.242901 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Enable Default Route ECMP" alias="enable-default-route-ecmp" description="Enables configurable ECMP behavior for default routes" updated="2023-02-27T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.243041 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: enable-default-route-ecmp Jun 01 11:23:06.243128 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: l3_ext_gw_mode.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.243356 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Neutron L3 Configurable external gateway mode" alias="ext-gw-mode" description="Extension of the router abstraction for specifying whether SNAT should occur on the external gateway" updated="2013-03-28T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.243498 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: ext-gw-mode Jun 01 11:23:06.243608 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: l3_ext_ha_mode.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.243825 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="HA Router extension" alias="l3-ha" description="Adds HA capability to routers." updated="2014-04-26T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.243974 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: l3-ha Jun 01 11:23:06.244067 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: l3_ext_ndp_proxy.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.244304 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Router support to enable NDP proxy" alias="l3-ext-ndp-proxy" description="Router extension to enable ndp proxy" updated="2020-08-25T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.244448 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension l3-ext-ndp-proxy not supported by any of loaded plugins Jun 01 11:23:06.244520 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: l3_extra_gws.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.244750 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Neutron L3 External Gateway Multihoming" alias="external-gateway-multihoming" description="Allow multiple external gateway ports per router" updated="2023-01-18T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.244894 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: external-gateway-multihoming Jun 01 11:23:06.244987 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: l3_flavors.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.245215 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Router Flavor Extension" alias="l3-flavors" description="Flavor support for routers." updated="2016-05-17T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.245354 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: l3-flavors Jun 01 11:23:06.245436 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: l3_ndp_proxy.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.245718 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="name" alias="l3-ndp-proxy" description="description" updated="2021-06-29T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.245862 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension l3-ndp-proxy not supported by any of loaded plugins Jun 01 11:23:06.245940 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: l3_port_ip_change_not_allowed.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.246191 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Prevent L3 router ports IP address change extension" alias="l3-port-ip-change-not-allowed" description="Prevent change of IP address for some L3 router ports" updated="2018-10-09T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.246328 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension l3-port-ip-change-not-allowed not supported by any of loaded plugins Jun 01 11:23:06.246409 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: l3agentscheduler.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.246871 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="L3 Agent Scheduler" alias="l3_agent_scheduler" description="Schedule routers among l3 agents" updated="2013-02-07T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.247023 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension l3_agent_scheduler not supported by any of loaded plugins Jun 01 11:23:06.247111 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: local_ip.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.247408 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Local IP" alias="local_ip" description="Support Node Local Virtual IP" updated="2021-07-26T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.247625 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension local_ip not supported by any of loaded plugins Jun 01 11:23:06.247696 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: logging.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.247971 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Logging API Extension" alias="logging" description="Provides a logging API for resources such as security group and firewall." updated="2017-01-01T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.248110 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension logging not supported by any of loaded plugins Jun 01 11:23:06.248202 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: metering.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.248615 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Neutron Metering" alias="metering" description="Neutron Metering extension." updated="2013-06-12T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.248763 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension metering not supported by any of loaded plugins Jun 01 11:23:06.248856 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: metering_source_and_destination_fields.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.249088 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Neutron Metering with source and destination filtering" alias="metering_source_and_destination_fields" description="Neutron Metering extension that enables the use of source and destination IP prefixes to create metering label rules." updated="2020-08-21T08:12:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.249247 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension metering_source_and_destination_fields not supported by any of loaded plugins Jun 01 11:23:06.249334 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: multiprovidernet.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.249688 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Multi Provider Network" alias="multi-provider" description="Expose mapping of virtual networks to multiple physical networks" updated="2013-06-27T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.249836 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: multi-provider Jun 01 11:23:06.249933 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: netmtu.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.250169 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Network MTU" alias="net-mtu" description="Provides MTU attribute for a network resource." updated="2015-03-25T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.250338 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: net-mtu Jun 01 11:23:06.250416 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: netmtu_writable.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.250660 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Network MTU (writable)" alias="net-mtu-writable" description="Provides a writable MTU attribute for a network resource." updated="2017-07-12T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.250801 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: net-mtu-writable Jun 01 11:23:06.250886 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: network_availability_zone.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.251138 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Network Availability Zone" alias="network_availability_zone" description="Availability zone support for network." updated="2015-01-01T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.251271 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: network_availability_zone Jun 01 11:23:06.251341 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: network_ha.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.251528 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Network HA creation flag" alias="network_ha" description="Network high availability creation flag." updated="2023-04-27T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.251691 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension network_ha not supported by any of loaded plugins Jun 01 11:23:06.251759 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: network_ip_availability.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.251960 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Network IP Availability" alias="network-ip-availability" description="Provides IP availability data for each network and subnet." updated="2015-09-24T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.252079 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: network-ip-availability Jun 01 11:23:06.252146 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: network_ip_availability_details.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.252350 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Network IP Availability Details Extension" alias="network-ip-availability-details" description="Add ip_availability_details attribute to network-ip-availabilities resource." updated="2025-10-31T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.252464 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: network-ip-availability-details Jun 01 11:23:06.252530 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: network_segment_range.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.252827 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Neutron Network Segment Range" alias="network-segment-range" description="Provides support for the network segment range management" updated="2018-11-29T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.252947 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension network-segment-range not supported by any of loaded plugins Jun 01 11:23:06.253011 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: pagination.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.253213 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Pagination support" alias="pagination" description="Extension that indicates that pagination is enabled." updated="2016-06-12T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.253282 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: pagination Jun 01 11:23:06.253348 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: port_device_profile.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.253523 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Port device profile" alias="port-device-profile" description="Expose the port device profile (Cyborg)" updated="2020-12-17T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.253672 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: port-device-profile Jun 01 11:23:06.253750 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: port_hardware_offload_type.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.253934 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Port hardware offload" alias="port-hardware-offload-type" description="Expose port hardware offload extension" updated="2023-05-09T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.254047 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension port-hardware-offload-type not supported by any of loaded plugins Jun 01 11:23:06.254112 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: port_hint_ovs_tx_steering.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.254364 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Port hint: Open vSwitch Tx steering" alias="port-hint-ovs-tx-steering" description="Enable Port hint: Open vSwitch Tx steering" updated="2023-01-01T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.254548 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension port-hint-ovs-tx-steering not supported by any of loaded plugins Jun 01 11:23:06.254659 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: port_hints.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.254887 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Port hints" alias="port-hints" description="Backend specific port hints to allow backend specific tuning" updated="2023-01-01T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.255039 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension port-hints not supported by any of loaded plugins Jun 01 11:23:06.255123 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: port_mac_address_override.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.255348 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Neutron Port MAC address override" alias="port-mac-override" description="Allow overriding the MAC address of a direct-physical Port via the active binding profile" updated="2022-03-18T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.255504 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension port-mac-override not supported by any of loaded plugins Jun 01 11:23:06.255608 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: port_mac_address_regenerate.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.255823 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Neutron Port MAC address regenerate" alias="port-mac-address-regenerate" description="Network port MAC address regenerate" updated="2018-05-03T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.255984 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: port-mac-address-regenerate Jun 01 11:23:06.256063 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: port_numa_affinity_policy.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.256297 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Port NUMA affinity policy" alias="port-numa-affinity-policy" description="Expose the port NUMA affinity policy" updated="2020-07-08T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.256465 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: port-numa-affinity-policy Jun 01 11:23:06.256578 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: port_numa_affinity_policy_socket.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.256762 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Port NUMA affinity policy "socket"" alias="port-numa-affinity-policy-socket" description="Adds "socket" to the supported port NUMA affinity policies" updated="2024-02-09T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.256880 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: port-numa-affinity-policy-socket Jun 01 11:23:06.256992 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: port_resource_request.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.257252 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Port Resource Request" alias="port-resource-request" description="Expose resource request to Port" updated="2018-05-08T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.257405 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension port-resource-request not supported by any of loaded plugins Jun 01 11:23:06.257509 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: port_resource_request_groups.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.257733 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Port Resource Request Groups" alias="port-resource-request-groups" description="Support requesting multiple groups of resources and traits from the same RP subtree in resource_request" updated="2021-08-02T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.257886 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension port-resource-request-groups not supported by any of loaded plugins Jun 01 11:23:06.257965 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: port_trusted_vif.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.258189 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Port trusted vif" alias="port-trusted-vif" description="Expose port 'trusted' attribute in the API" updated="2024-07-10T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.258336 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension port-trusted-vif not supported by any of loaded plugins Jun 01 11:23:06.258416 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: portbindings.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.258657 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Port Binding" alias="binding" description="Expose port bindings of a virtual port to external application" updated="2014-02-03T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.258802 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: binding Jun 01 11:23:06.258884 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: portbindings_extended.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.259148 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Port Bindings Extended" alias="binding-extended" description="Expose port bindings of a virtual port to external application" updated="2017-07-17T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.259331 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: binding-extended Jun 01 11:23:06.259447 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: portsecurity.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.259727 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Port Security" alias="port-security" description="Provides port security" updated="2012-07-23T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.259896 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: port-security Jun 01 11:23:06.259993 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: project_id.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.260252 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="project_id field enabled" alias="project-id" description="Extension that indicates that project_id field is enabled." updated="2016-09-09T09:09:09-09:09" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.260355 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: project-id Jun 01 11:23:06.260456 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: providernet.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.260732 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Provider Network" alias="provider" description="Expose mapping of virtual networks to physical networks" updated="2012-09-07T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.260920 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: provider Jun 01 11:23:06.261020 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: pvlan.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.261275 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Private VLAN" alias="pvlan" description="The Private VLAN semantics extension." updated="2026-02-03T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.261453 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension pvlan not supported by any of loaded plugins Jun 01 11:23:06.261571 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: qinq.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.261853 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="QinQ networks" alias="qinq" description="Expose network 'qinq' attribute in the API. This allows to configure network to allow vlan in vlan configuration using 0x88a8 ethertype (QinQ)." updated="2024-12-02T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.262029 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: qinq Jun 01 11:23:06.262130 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: qos.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.263066 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Quality of Service" alias="qos" description="The Quality of Service extension." updated="2015-06-08T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.263266 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension qos not supported by any of loaded plugins Jun 01 11:23:06.263369 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: qos_bw_limit_direction.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.263691 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Direction for QoS bandwidth limit rule" alias="qos-bw-limit-direction" description="Allow to configure QoS bandwidth limit rule with specific direction: ingress or egress" updated="2017-04-10T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.263876 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension qos-bw-limit-direction not supported by any of loaded plugins Jun 01 11:23:06.263982 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: qos_bw_minimum_ingress.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.264247 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Ingress direction for QoS minimum bandwidth rule" alias="qos-bw-minimum-ingress" description="Allow to configure QoS minumum bandwidth rule with ingress direction." updated="2018-07-09T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.264427 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension qos-bw-minimum-ingress not supported by any of loaded plugins Jun 01 11:23:06.264533 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: qos_default.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.264805 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="QoS default policy" alias="qos-default" description="Expose the QoS default policy per project" updated="2017-041-06T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.264978 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension qos-default not supported by any of loaded plugins Jun 01 11:23:06.265072 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: qos_fip.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.265335 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Floating IP QoS" alias="qos-fip" description="The floating IP Quality of Service extension" updated="2017-07-20T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.265503 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension qos-fip not supported by any of loaded plugins Jun 01 11:23:06.265624 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: qos_fip_network_policy.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.265872 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="QoS floating IP network policy ID" alias="qos-fip-network-policy" description="Adds a the QoS network ID to the floating IP definition" updated="2021-11-15T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.266042 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension qos-fip-network-policy not supported by any of loaded plugins Jun 01 11:23:06.266141 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: qos_gateway_ip.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.266394 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Router gateway IP QoS" alias="qos-gateway-ip" description="The Router gateway IP Quality of Service extension" updated="2018-02-24T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.266590 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension qos-gateway-ip not supported by any of loaded plugins Jun 01 11:23:06.266693 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: qos_port_network_policy.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.266932 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="QoS port network policy ID" alias="qos-port-network-policy" description="Adds a the QoS network ID to the port definition" updated="2019-11-01T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.267104 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension qos-port-network-policy not supported by any of loaded plugins Jun 01 11:23:06.267214 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: qos_pps_minimum_rule.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.267483 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="QoS minimum packet rate rule" alias="qos-pps-minimum" description="Allow to configure QoS minimum packet rate rule." updated="2021-07-14T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.267679 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension qos-pps-minimum not supported by any of loaded plugins Jun 01 11:23:06.267785 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: qos_pps_minimum_rule_alias.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.268047 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="QoS minimum packet rate rule alias" alias="qos-pps-minimum-rule-alias" description="API to enable GET, PUT and DELETE operations on QoS minimum packet rate rule without specifying policy ID" updated="2021-10-22T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.268234 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension qos-pps-minimum-rule-alias not supported by any of loaded plugins Jun 01 11:23:06.268333 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: qos_pps_rule.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.268635 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="QoS Rule Type Packet per Second Extension" alias="qos-pps" description="Add QoS Rule Type Packet per Second" updated="2021-05-12T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.268812 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension qos-pps not supported by any of loaded plugins Jun 01 11:23:06.268915 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: qos_rule_type_details.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.269155 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Details of QoS rule types" alias="qos-rule-type-details" description="Expose details about QoS rule types supported by loaded backend drivers" updated="2017-06-22T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.269341 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension qos-rule-type-details not supported by any of loaded plugins Jun 01 11:23:06.269439 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: qos_rule_type_filter.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.269707 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Allow to filter the list of QoS rule types" alias="qos-rule-type-filter" description="Allows to filter the QoS rule type list adding two new flags. "all_rules" prints all implemented QoS rule types."all_supported" prints all supported QoS rule types by the loaded mechanism drivers" updated="2022-02-02T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.269881 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension qos-rule-type-filter not supported by any of loaded plugins Jun 01 11:23:06.269980 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: qos_rules_alias.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.270266 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Quality of Service rules alias API" alias="qos-rules-alias" description="API to enable GET, PUT and DELETE operations on QoS policy rules without specifying policy ID" updated="2018-10-07T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.270429 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension qos-rules-alias not supported by any of loaded plugins Jun 01 11:23:06.270521 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: quota_check_limit.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.270817 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Quota engine limit check" alias="quota-check-limit" description="Support for checking the resource usage before applying a new quota limit" updated="2021-09-08T16:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.271001 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: quota-check-limit Jun 01 11:23:06.271104 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: quota_check_limit_default.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.271368 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Quota engine limit check by default" alias="quota-check-limit-default" description="By default, the Neutron quota engine checks the resource usage before applying a new quota limit" updated="2024-08-21T16:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.271556 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: quota-check-limit-default Jun 01 11:23:06.271679 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: quotasv2.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.272257 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Quota management support" alias="quotas" description="Expose functions for quotas management per project" updated="2012-07-29T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.272365 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: quotas Jun 01 11:23:06.272472 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: quotasv2_detail.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.273129 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Quota details management support" alias="quota_details" description="Expose functions for quotas usage statistics per project" updated="2017-02-10T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.273252 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: quota_details Jun 01 11:23:06.273354 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: rbac.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.273951 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="RBAC Policies" alias="rbac-policies" description="Allows creation and modification of policies that control project access to resources." updated="2015-06-17T12:15:12-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.274142 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: rbac-policies Jun 01 11:23:06.274251 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: rbac_address_group.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.274518 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Add address_group type to RBAC" alias="rbac-address-group" description="Add address_group type to network RBAC" updated="2021-01-20T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.274722 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension rbac-address-group not supported by any of loaded plugins Jun 01 11:23:06.274821 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: rbac_address_scope.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.275079 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Add address_scope type to RBAC" alias="rbac-address-scope" description="Add address_scope type to RBAC" updated="2020-02-12T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.275258 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: rbac-address-scope Jun 01 11:23:06.275356 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: rbac_security_groups.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.275631 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Add security_group type to network RBAC" alias="rbac-security-groups" description="Add security_group type to network RBAC" updated="2019-02-14T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.275807 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: rbac-security-groups Jun 01 11:23:06.276225 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: rbac_subnetpool.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.276225 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Add subnetpool type to RBAC" alias="rbac-subnetpool" description="Add subnetpool type to RBAC" updated="2020-02-05T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.276336 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension rbac-subnetpool not supported by any of loaded plugins Jun 01 11:23:06.276437 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: revisionifmatch.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.276707 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="If-Match constraints based on revision_number" alias="revision-if-match" description="Extension indicating that If-Match based on revision_number is supported." updated="2016-12-11T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.276881 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: revision-if-match Jun 01 11:23:06.276985 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: revisions.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.277269 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Resource revision numbers" alias="standard-attr-revisions" description="This extension will display the revision number of neutron resources." updated="2016-04-11T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.277450 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: standard-attr-revisions Jun 01 11:23:06.277595 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: router_availability_zone.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.277876 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Router Availability Zone" alias="router_availability_zone" description="Availability zone support for router." updated="2015-01-01T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.278055 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: router_availability_zone Jun 01 11:23:06.278155 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: router_enable_snat.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.278417 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Exposes the router "enable_snat" read-only configuration flag" alias="router-enable-snat" description="Exposes the router "enable_snat" read-only configuration flag" updated="2025-09-18T18:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.278601 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: router-enable-snat Jun 01 11:23:06.278697 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: routerservicetype.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.278924 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Router Service Type" alias="router-service-type" description="Provides router service type" updated="2013-01-29T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.279047 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension router-service-type not supported by any of loaded plugins Jun 01 11:23:06.279114 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: security_groups_default_rules.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.279519 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Default rules for security groups" alias="security-groups-default-rules" description="Configure set of security group rules used as default rules for every new security group" updated="2022-12-19T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.279701 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: security-groups-default-rules Jun 01 11:23:06.279799 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: security_groups_normalized_cidr.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.280016 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Normalized CIDR field for security group rules" alias="security-groups-normalized-cidr" description="Add new field with normalized remote_ip_prefix cidr in SG rule" updated="2020-07-28T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.280198 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: security-groups-normalized-cidr Jun 01 11:23:06.280283 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: security_groups_port_filtering.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.280504 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Port filtering on security groups" alias="port-security-groups-filtering" description="Provides security groups filtering when listing ports" updated="2018-01-09T09:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.280688 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension port-security-groups-filtering not supported by any of loaded plugins Jun 01 11:23:06.280784 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: security_groups_remote_address_group.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.281015 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Remote address group id field for security group rules" alias="security-groups-remote-address-group" description="Add new field of remote address group id in SG rules" updated="2020-08-25T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.281169 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: security-groups-remote-address-group Jun 01 11:23:06.281277 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: security_groups_rules_belongs_to_default_sg.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.281489 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Security group rule belongs to the project's default security group" alias="security-groups-rules-belongs-to-default-sg" description="Flag to determine if the security group rule belongs to the project's default security group" updated="2023-05-23T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.281669 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: security-groups-rules-belongs-to-default-sg Jun 01 11:23:06.281766 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: security_groups_shared_filtering.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.281985 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Security group filtering on the shared field" alias="security-groups-shared-filtering" description="Support filtering security groups on the shared field" updated="2021-10-05T09:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.282154 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: security-groups-shared-filtering Jun 01 11:23:06.282262 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: securitygroup.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.283671 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="security-group" alias="security-group" description="The security groups extension." updated="2012-10-05T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.283841 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: security-group Jun 01 11:23:06.283943 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: segment.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.284235 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Segment" alias="segment" description="Segments extension." updated="2016-02-24T17:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.284392 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension segment not supported by any of loaded plugins Jun 01 11:23:06.284492 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: segments_peer_subnet_host_routes.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.284729 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Segments peer-subnet host routes" alias="segments-peer-subnet-host-routes" description="Add host routes to subnets on a routed network (segments)" updated="2018-06-12T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.284888 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension segments-peer-subnet-host-routes not supported by any of loaded plugins Jun 01 11:23:06.284989 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: servicetype.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.285238 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Neutron Service Type Management" alias="service-type" description="API for retrieving service providers for Neutron advanced services" updated="2013-01-20T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.285392 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: service-type Jun 01 11:23:06.285497 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: sorting.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.285730 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Sorting support" alias="sorting" description="Extension that indicates that sorting is enabled." updated="2016-06-12T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.285830 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: sorting Jun 01 11:23:06.285932 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: standard_attr_segment.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.286153 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Standard Attribute Segment Extension" alias="standard-attr-segment" description="Add standard attributes to Segment resource" updated="2018-04-09T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.286333 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension standard-attr-segment not supported by any of loaded plugins Jun 01 11:23:06.286429 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: standardattrdescription.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.286710 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="standard-attr-description" alias="standard-attr-description" description="Extension to add descriptions to standard attributes" updated="2016-02-10T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.286860 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: standard-attr-description Jun 01 11:23:06.286953 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: stateful_security_group.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.287167 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Stateful security group" alias="stateful-security-group" description="Indicates if the security group is stateful or not" updated="2019-11-26T09:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.287341 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: stateful-security-group Jun 01 11:23:06.287420 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: stdattrs_common.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.287754 np0000001841 neutron-ovn-maintenance-worker[88598]: WARNING neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Did not find expected name "Stdattrs_common" in /opt/stack/neutron/neutron/extensions/stdattrs_common.py Jun 01 11:23:06.287828 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: subnet_dns_publish_fixed_ip.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.288022 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Subnet DNS publish fixed IP" alias="subnet-dns-publish-fixed-ip" description="Support choosing to publish DNS records for IPs from a subnet" updated="2019-05-24T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.288146 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: subnet-dns-publish-fixed-ip Jun 01 11:23:06.288227 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: subnet_external_network.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.288422 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Subnet belongs to an external network" alias="subnet-external-network" description="Informs if the subnet belongs to an external network" updated="2024-02-05T18:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.288534 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: subnet-external-network Jun 01 11:23:06.288633 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: subnet_onboard.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.288837 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Subnet Onboard" alias="subnet_onboard" description="Provides support for onboarding subnets into subnet pools" updated="2018-12-18T09:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.288955 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension subnet_onboard not supported by any of loaded plugins Jun 01 11:23:06.289022 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: subnet_segmentid_writable.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.289216 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Subnet SegmentID (writable)" alias="subnet-segmentid-writable" description="Provides a writable segment_id attribute for a subnet resource." updated="2018-03-12T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.289332 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension subnet-segmentid-writable not supported by any of loaded plugins Jun 01 11:23:06.289398 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: subnet_service_types.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.289801 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Subnet service types" alias="subnet-service-types" description="Provides ability to set the subnet service_types field" updated="2016-03-15T18:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.289931 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: subnet-service-types Jun 01 11:23:06.290004 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: subnetallocation.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.290220 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Subnet Allocation" alias="subnet_allocation" description="Enables allocation of subnets from a subnet pool" updated="2015-03-30T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.290342 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: subnet_allocation Jun 01 11:23:06.290414 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: subnetpool_prefix_ops.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.290660 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Subnet Pool Prefix Operations" alias="subnetpool-prefix-ops" description="Provides support for adjusting the prefix list of subnet pools" updated="2019-02-08T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.290780 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: subnetpool-prefix-ops Jun 01 11:23:06.290847 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: tag_creation.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.291025 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Tag creation extension" alias="tag-creation" description="Allow to create multiple tags for a resource" updated="2024-23-07T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.291142 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: tag-creation Jun 01 11:23:06.291231 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: tag_ports_during_bulk_creation.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.291410 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Tag Ports During Bulk Creation" alias="tag-ports-during-bulk-creation" description="Allow to tag ports during bulk creation" updated="2019-12-29T19:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.291551 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension tag-ports-during-bulk-creation not supported by any of loaded plugins Jun 01 11:23:06.291630 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: tagging.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.292328 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Tag support for resources with standard attribute: port, subnet, subnetpool, network, security_group, router, floatingip, policy, trunk, network_segment_range" alias="standard-attr-tag" description="Enables to set tag on resources with standard attribute." updated="2017-01-01T00:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.292473 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: standard-attr-tag Jun 01 11:23:06.292552 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: timestamp.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.292768 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Resource timestamps" alias="standard-attr-timestamp" description="Adds created_at and updated_at fields to all Neutron resources that have Neutron standard attributes." updated="2016-09-12T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.292889 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: standard-attr-timestamp Jun 01 11:23:06.292952 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: trunk.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.293150 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Trunk Extension" alias="trunk" description="Provides support for trunk ports" updated="2016-01-01T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.293294 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension trunk not supported by any of loaded plugins Jun 01 11:23:06.293362 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: trunk_details.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.293580 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Trunk port details" alias="trunk-details" description="Expose trunk port details" updated="2016-01-01T10:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.293697 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension trunk-details not supported by any of loaded plugins Jun 01 11:23:06.293764 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: uplink_status_propagation.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.293938 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Uplink status propagation extension" alias="uplink-status-propagation" description="Provides ability to propagate port uplink status." updated="2018-05-31T18:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.294053 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension uplink-status-propagation not supported by any of loaded plugins Jun 01 11:23:06.294120 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: uplink_status_propagation_updatable.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.294316 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Uplink status propagation updatable extension" alias="uplink-status-propagation-updatable" description="Provides ability to update the port uplink status." updated="2024-09-03T18:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.294450 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Extension uplink-status-propagation-updatable not supported by any of loaded plugins Jun 01 11:23:06.294522 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loading extension file: vlantransparent.py {{(pid=88598) _load_all_extensions_from_path /opt/stack/neutron/neutron/api/extensions.py:449}} Jun 01 11:23:06.294793 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Ext name="Vlantransparent" alias="vlan-transparent" description="Provides Vlan Transparent Networks" updated="2015-03-23T09:00:00-00:00" {{(pid=88598) _check_extension /opt/stack/neutron/neutron/api/extensions.py:415}} Jun 01 11:23:06.294948 np0000001841 neutron-ovn-maintenance-worker[88598]: INFO neutron.api.extensions [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Loaded extension: vlan-transparent Jun 01 11:23:06.299015 np0000001841 neutron-ovn-maintenance-worker[88598]: WARNING oslo_service.backend._threading.service [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Service MaintenanceWorker is not picklable with spawn; falling back to fork. Please make the service spawn-safe to avoid this fallback. Jun 01 11:23:06.299344 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Full set of CONF: {{(pid=88598) _load_service_manager_options /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/oslo_config_glue.py:59}} Jun 01 11:23:06.299422 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ******************************************************************************** {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3229}} Jun 01 11:23:06.299499 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Configuration options gathered from: {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3230}} Jun 01 11:23:06.299588 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] command line args: ['--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3231}} Jun 01 11:23:06.299691 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] config files: ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3232}} Jun 01 11:23:06.299752 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ================================================================================ {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3237}} Jun 01 11:23:06.299862 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] agent_down_time = 75 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.299942 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] allow_automatic_dhcp_failover = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.300022 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] allow_bulk = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.300196 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] allowed_conntrack_helpers = [{'amanda': 'tcp'}, {'ftp': 'tcp'}, {'h323': 'udp'}, {'h323': 'tcp'}, {'irc': 'tcp'}, {'netbios-ns': 'udp'}, {'pptp': 'tcp'}, {'sane': 'tcp'}, {'sip': 'udp'}, {'sip': 'tcp'}, {'snmp': 'udp'}, {'tftp': 'udp'}] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.300293 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] api_extensions_path = {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.300377 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] api_paste_config = api-paste.ini {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.300480 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] api_workers = 2 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.300592 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] auth_strategy = keystone {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.300676 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] base_mac = fa:16:3e:00:00:00 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.300778 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] bind_host = 0.0.0.0 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.300862 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] bind_port = 9696 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.300940 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] client_socket_timeout = 900 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.301026 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] config_dir = [] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.301086 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] config_file = ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.301166 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] config_source = [] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.301261 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] control_exchange = neutron {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.301357 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] core_plugin = ml2 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.301466 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] debug = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.301562 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] default_availability_zones = [] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.301700 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] default_log_levels = ['amqp=WARN', 'boto=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'OFPHandler=INFO', 'OfctlService=INFO', 'os_ken.base.app_manager=INFO', 'os_ken.controller.controller=INFO'] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.301782 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] dhcp_agent_notification = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.301860 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] dhcp_agents_per_network = 1 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.301937 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] dhcp_lease_duration = 86400 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.302018 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] dhcp_load_type = networks {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.302096 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] dns_domain = openstacklocal {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.302183 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] dvr_base_mac = fa:16:3f:00:00:00 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.302272 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] enable_default_route_bfd = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.302363 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] enable_default_route_ecmp = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.302458 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] enable_dvr = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.302548 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] enable_new_agents = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.302636 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] enable_services_on_agents_with_admin_state_down = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.302711 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] enable_snat_by_default = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.302786 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] enable_traditional_dhcp = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.302859 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] external_dns_driver = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.302932 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] filter_validation = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.303013 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] global_physnet_mtu = 1430 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.303070 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] graceful_shutdown_timeout = 60 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.303206 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] host = np0000001841 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.303295 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] host_dvr_for_dhcp = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.303403 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] http_retries = 3 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.303501 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] instance_format = [instance: %(uuid)s] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.303607 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] instance_uuid_format = [instance: %(uuid)s] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.303688 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ipam_driver = internal {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.303765 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ipv6_pd_enabled = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.303839 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] l3_ha = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.303922 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] l3_ha_net_cidr = 169.254.192.0/18 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.304031 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] l3_ha_network_physical_name = {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.304061 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] l3_ha_network_type = {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.304146 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] log_color = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.304244 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] log_config_append = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.304327 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.304440 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] log_dir = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.304522 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] log_file = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.304597 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] log_options = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.304678 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] log_rotate_interval = 1 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.304759 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] log_rotate_interval_type = days {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.304839 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] log_rotation_type = none {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.304897 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.304955 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.305037 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.305117 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.305183 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.305263 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] max_allowed_address_pair = 10 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.305350 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] max_dns_nameservers = 5 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.305465 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] max_header_line = 16384 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.305631 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] max_l3_agents_per_router = 3 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.305703 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] max_logfile_count = 30 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.305810 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] max_logfile_size_mb = 200 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.305890 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] max_routes = 30 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.305967 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] max_subnet_host_routes = 20 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.306051 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] my_ip = 10.4.3.71 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.306129 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] my_ipv6 = ::1 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.306221 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] network_auto_schedule = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.306298 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] network_link_prefix = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.306397 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.WeightScheduler {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.306507 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] notify_nova_on_port_data_changes = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.306610 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] notify_nova_on_port_status_changes = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.306691 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] pagination_max_limit = -1 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.306783 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] periodic_fuzzy_delay = 5 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.306862 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] periodic_interval = 40 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.306936 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] publish_errors = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.307009 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] rate_limit_burst = 0 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.307090 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] rate_limit_except_level = CRITICAL {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.307165 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] rate_limit_interval = 0 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.307251 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] router_distributed = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.307329 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] rpc_resources_processing_step = 20 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.307409 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] rpc_response_max_timeout = 600 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.307501 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] rpc_response_timeout = 60 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.307608 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] rpc_state_report_workers = 0 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.307685 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] rpc_workers = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.307775 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] send_events_interval = 2 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.307864 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] service_plugins = ['ovn-router'] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.307945 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] setproctitle = on {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.308019 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] shell_completion = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.308097 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] state_path = /opt/stack/data/neutron {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.308184 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] syslog_log_facility = LOG_USER {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.308270 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] tcp_keepidle = 600 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.308351 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] transport_url = **** {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.308441 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] use_journal = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.308523 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] use_json = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.308618 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] use_stderr = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.308698 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] use_syslog = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.308786 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] vlan_qinq = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.308863 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] vlan_transparent = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.308937 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] wsgi_default_pool_size = 100 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.309011 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] wsgi_keep_alive = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.309092 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.309166 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] wsgi_server_debug = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.309272 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_concurrency.disable_process_locking = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.309360 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_concurrency.lock_path = /opt/stack/data/neutron/lock {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.309469 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] profiler.connection_string = messaging:// {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.309568 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] profiler.enabled = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.309661 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] profiler.es_doc_type = notification {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.309743 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] profiler.es_scroll_size = 10000 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.309831 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] profiler.es_scroll_time = 2m {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.309909 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] profiler.filter_error_trace = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.309990 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] profiler.hmac_keys = **** {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.310070 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] profiler.sentinel_service_name = mymaster {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.310156 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] profiler.socket_timeout = 0.1 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.310249 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] profiler.trace_requests = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.310325 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] profiler.trace_sqlalchemy = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.310416 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] profiler_jaeger.process_tags = {} {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.310503 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] profiler_jaeger.service_name_prefix = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.310602 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] profiler_otlp.service_name_prefix = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.310687 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.310795 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_metrics.metrics_enabled = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.310909 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_metrics.metrics_process_name = {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.310995 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.311073 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.311161 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_policy.enforce_new_defaults = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.311385 np0000001841 neutron-ovn-maintenance-worker[88598]: WARNING oslo_config.cfg [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Deprecated: Option "enforce_scope" from group "oslo_policy" is deprecated for removal (This configuration was added temporarily to facilitate a smooth transition to the new RBAC. OpenStack will always enforce scope checks. This configuration option is deprecated and will be removed in the 2025.2 cycle.). Its value may be silently ignored in the future. Jun 01 11:23:06.311494 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_policy.enforce_scope = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.311603 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_policy.policy_default_rule = default {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.311698 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_policy.policy_dirs = ['policy.d'] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.311798 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_policy.policy_file = /etc/neutron/policy.json {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.311884 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.311959 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.312035 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_policy.remote_ssl_client_crt_file = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.312109 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_policy.remote_ssl_client_key_file = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.312201 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.312285 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_policy.remote_timeout = 60.0 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.312387 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.312488 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] service_providers.service_provider = [] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.312599 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.auth_section = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.312690 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.auth_type = password {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.312787 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.auth_url = https://10.4.3.71/identity {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.312868 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.cafile = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.312946 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.certfile = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.313026 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.collect_timing = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.313102 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.default_domain_id = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.313185 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.default_domain_name = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.313262 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.domain_id = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.313347 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.domain_name = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.313459 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.endpoint_type = public {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.313578 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.insecure = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.313659 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.keyfile = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.313755 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.password = **** {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.313836 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.project_domain_id = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.313919 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.project_domain_name = Default {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.313999 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.project_id = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.314083 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.project_name = service {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.314165 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.region_name = RegionOne {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.314261 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.split_loggers = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.314345 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.system_scope = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.314441 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.timeout = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.314523 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.trust_id = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.314619 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.user_domain_id = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.314701 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.user_domain_name = Default {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.314798 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.user_id = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.314889 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] nova.username = nova {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.314971 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] placement.auth_section = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.315056 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] placement.auth_type = password {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.315132 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] placement.cafile = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.315220 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] placement.certfile = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.315297 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] placement.collect_timing = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.315409 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] placement.endpoint_type = public {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.315501 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] placement.insecure = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.315594 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] placement.keyfile = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.315699 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] placement.region_name = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.315817 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] placement.split_loggers = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.315902 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] placement.timeout = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.315985 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.auth_section = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.316062 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.auth_type = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.316136 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.cafile = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.316229 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.certfile = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.316311 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.collect_timing = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.316413 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.connect_retries = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.316505 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.connect_retry_delay = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.316603 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.enable_notifications = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.316682 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.endpoint_override = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.316757 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.insecure = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.316831 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.interface = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.316903 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.keyfile = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.316976 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.max_version = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.317048 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.min_version = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.317122 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.region_name = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.317212 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.retriable_status_codes = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.317291 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.service_name = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.317365 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.service_type = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.317441 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.split_loggers = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.317548 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.status_code_retries = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.317651 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.status_code_retry_delay = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.317718 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.timeout = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.317795 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.valid_interfaces = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.317869 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ironic.version = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.317949 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] cli_script.dry_run = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.318035 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.318112 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.318218 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.318303 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.318381 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.318457 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.318534 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.318633 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.318720 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.hostname = np0000001841 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.318796 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.kombu_compression = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.318878 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.318958 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.319037 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.319115 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.kombu_reconnect_splay = 0.0 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.319206 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.processname = neutron-ovn-maintenance-worker {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.319283 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.319359 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.319441 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.319517 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.319612 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.319691 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.319775 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.319842 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.319916 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.319989 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.320063 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.320139 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.320238 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.320315 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.320389 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.ssl = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.320470 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.ssl_ca_file = {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.320562 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.ssl_cert_file = {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.320644 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.320719 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.ssl_enforce_hostname_verification = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.320797 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.ssl_key_file = {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.320874 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.ssl_version = {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.320981 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_rabbit.use_queue_manager = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.321069 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_notifications.driver = [] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.321147 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_notifications.retry = -1 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.321256 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_notifications.topics = ['notifications'] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.321335 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_messaging_notifications.transport_url = **** {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.321415 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_reports.file_event_handler = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.321494 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_reports.file_event_handler_interval = 1 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.321607 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_reports.log_dir = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.321692 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.321775 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] QUOTAS.default_quota = -1 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.321855 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] QUOTAS.quota_driver = neutron.db.quota.driver_nolock.DbQuotaNoLockDriver {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.321932 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] QUOTAS.quota_floatingip = 50 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.322008 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] QUOTAS.quota_network = 100 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.322083 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] QUOTAS.quota_port = 500 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.322157 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] QUOTAS.quota_rbac_policy = 10 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.322250 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] QUOTAS.quota_router = 10 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.322326 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] QUOTAS.quota_security_group = 10 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.322401 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] QUOTAS.quota_security_group_rule = 100 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.322475 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] QUOTAS.quota_subnet = 100 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.322566 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] QUOTAS.track_quota_usage = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.322654 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep.capabilities = [21, 12, 1, 2, 19] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.322731 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep.group = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.322806 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep.helper_command = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.322881 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep.log_daemon_traceback = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.322960 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep.logger_name = oslo_privsep.daemon {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.323038 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep.thread_pool_size = 12 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.323111 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep.user = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.323201 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_dhcp_release.capabilities = [21, 12] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.323275 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_dhcp_release.group = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.323348 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_dhcp_release.helper_command = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.323422 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_dhcp_release.log_daemon_traceback = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.323501 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_dhcp_release.logger_name = oslo_privsep.daemon {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.323596 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_dhcp_release.thread_pool_size = 12 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.323668 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_dhcp_release.user = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.323746 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_ovs_vsctl.capabilities = [21, 12] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.323818 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_ovs_vsctl.group = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.323890 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_ovs_vsctl.helper_command = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.323962 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_ovs_vsctl.log_daemon_traceback = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.324039 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_ovs_vsctl.logger_name = oslo_privsep.daemon {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.324112 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_ovs_vsctl.thread_pool_size = 12 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.324195 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_ovs_vsctl.user = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.324284 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_namespace.capabilities = [21] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.324357 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_namespace.group = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.324427 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_namespace.helper_command = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.324502 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_namespace.log_daemon_traceback = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.324597 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_namespace.logger_name = oslo_privsep.daemon {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.324670 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_namespace.thread_pool_size = 12 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.324741 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_namespace.user = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.324818 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_conntrack.capabilities = [12] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.324892 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_conntrack.group = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.324963 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_conntrack.helper_command = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.325037 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_conntrack.log_daemon_traceback = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.325113 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_conntrack.logger_name = oslo_privsep.daemon {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.325199 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_conntrack.thread_pool_size = 12 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.325279 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_conntrack.user = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.325364 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_link.capabilities = [12, 21] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.325435 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_link.group = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.325523 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_link.helper_command = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.325633 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_link.log_daemon_traceback = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.325717 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_link.logger_name = oslo_privsep.daemon {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.325792 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_link.thread_pool_size = 12 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.325876 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] privsep_link.user = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.326018 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] AGENT.root_helper = sudo /opt/stack/data/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.326073 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] AGENT.root_helper_daemon = sudo /opt/stack/data/venv/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.326149 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] AGENT.use_helper_for_ns_read = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.326251 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] SECURITYGROUP.enable_ipset = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.326335 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] SECURITYGROUP.enable_security_group = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.326408 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] SECURITYGROUP.firewall_driver = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.326487 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] SECURITYGROUP.permitted_ethertypes = [] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.326589 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] experimental.ipv6_pd_enabled = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.326685 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ml2.extension_drivers = ['port_security', 'qos'] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.326761 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ml2.external_network_type = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.326844 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ml2.mechanism_drivers = ['ovn'] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.326927 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ml2.overlay_ip_version = 4 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.327003 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ml2.path_mtu = 0 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.327081 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ml2.physical_network_mtus = {} {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.327165 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ml2.project_network_types = ['geneve'] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.327258 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ml2.tunnelled_network_rp_name = rp_tunnelled {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.327344 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ml2.type_drivers = ['local', 'flat', 'vlan', 'geneve'] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.327434 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ml2_type_flat.flat_networks = ['public'] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.327518 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ml2_type_geneve.max_header_size = 38 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.327619 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ml2_type_geneve.vni_ranges = ['1:65536'] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.327705 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ml2_type_vlan.network_vlan_ranges = ['public'] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.327785 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] network_log.burst_limit = 25 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.327860 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] network_log.local_output_log_base = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.327935 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] network_log.rate_limit = 100 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.328014 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.asyncio_connection = **** {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.328088 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.asyncio_slave_connection = **** {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.328167 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.backend = sqlalchemy {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.328262 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.connection = **** {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.328337 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.connection_debug = 0 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.328416 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.connection_parameters = {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.328493 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.connection_recycle_time = 3600 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.328587 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.connection_trace = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.328665 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.db_inc_retry_interval = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.328740 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.db_max_retries = 20 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.328815 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.db_max_retry_interval = 10 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.328888 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.db_retry_interval = 1 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.328964 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.max_overflow = 50 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.329037 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.max_pool_size = 5 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.329112 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.max_retries = 10 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.329203 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.mysql_sql_mode = TRADITIONAL {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.329278 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.mysql_wsrep_sync_wait = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.329350 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.pool_timeout = None {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.329424 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.retry_interval = 10 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.329498 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.slave_connection = **** {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.329607 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.sqlite_synchronous = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.329684 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.synchronous_reader = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.329758 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] database.use_db_reconnect = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.329849 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.bfd_min_rx = 1000 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.329921 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.bfd_min_tx = 100 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.329995 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.bfd_mult = 3 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.330070 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.broadcast_arps_to_all_routers = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.330145 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.dhcp_default_lease_time = 43200 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.330233 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.disable_ovn_dhcp_for_baremetal_ports = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.330309 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.dns_records_ovn_owned = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.330387 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.dns_servers = [] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.330464 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.enable_distributed_floating_ip = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.330549 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.fdb_age_threshold = 300 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.330646 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ha_failover_strategy = normal {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.330727 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.live_migration_activation_strategy = rarp {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.330802 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.localnet_learn_fdb = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.330876 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.mac_binding_age_threshold = 300 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.330957 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.neutron_sync_mode = log {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.331062 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovn_dhcp4_global_options = {} {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.331149 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovn_dhcp6_global_options = {} {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.331246 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovn_l3_scheduler = leastloaded {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.331327 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovn_metadata_enabled = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.331406 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovn_nb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.331485 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovn_nb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.331590 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovn_nb_connection = ['ssl:10.4.3.71:6641'] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.331672 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovn_nb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.331748 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovn_router_indirect_snat = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.331827 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovn_sb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.331905 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovn_sb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.331988 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovn_sb_connection = ['ssl:10.4.3.71:6642'] {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.332066 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovn_sb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.332140 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovs_create_tap = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.332231 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovsdb_connection_timeout = 180 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.332313 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovsdb_log_level = INFO {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.332388 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovsdb_probe_interval = 60000 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.332464 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.ovsdb_retry_max_interval = 180 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.332545 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.stateless_nat_enabled = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.332641 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn.vhost_sock_dir = /var/run/openvswitch {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.332726 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] OVS.bridge_mac_table_size = 50000 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.332804 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] OVS.igmp_flood = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.332883 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] OVS.igmp_flood_reports = True {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.332962 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] OVS.igmp_flood_unregistered = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.333044 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] OVS.igmp_snooping_enable = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.333118 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] OVS.ovsdb_timeout = 10 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.333210 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn_nb_global.fdb_removal_limit = 0 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.333287 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn_nb_global.ignore_lsp_down = False {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.333363 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ovn_nb_global.mac_binding_removal_limit = 0 {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.333420 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG cotyledon.oslo_config_glue [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] ******************************************************************************** {{(pid=88598) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3263}} Jun 01 11:23:06.333515 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG oslo_service.backend._threading.service [None req-c5aa4541-6e83-4259-8c18-8224148d89dc None None] Selected the multiprocessing context: and updated it in Cotyledon ServiceManager: {{(pid=88598) launch_service /opt/stack/data/venv/lib/python3.12/site-packages/oslo_service/backend/_threading/service.py:405}} Jun 01 11:23:06.352523 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] Full set of CONF: {{(pid=89427) _load_service_options /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/oslo_config_glue.py:65}} Jun 01 11:23:06.352523 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3229}} Jun 01 11:23:06.353055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] Configuration options gathered from: {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3230}} Jun 01 11:23:06.353055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] command line args: ['--config-file', '/etc/neutron/neutron.conf', '--config-file', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3231}} Jun 01 11:23:06.353055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] config files: ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3232}} Jun 01 11:23:06.353055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ================================================================================ {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3237}} Jun 01 11:23:06.353155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] agent_down_time = 75 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.353231 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] allow_automatic_dhcp_failover = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.353313 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] allow_bulk = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.353506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] allowed_conntrack_helpers = [{'amanda': 'tcp'}, {'ftp': 'tcp'}, {'h323': 'udp'}, {'h323': 'tcp'}, {'irc': 'tcp'}, {'netbios-ns': 'udp'}, {'pptp': 'tcp'}, {'sane': 'tcp'}, {'sip': 'udp'}, {'sip': 'tcp'}, {'snmp': 'udp'}, {'tftp': 'udp'}] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.353605 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] api_extensions_path = {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.353678 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] api_paste_config = api-paste.ini {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.353777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] api_workers = 2 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.353857 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] auth_strategy = keystone {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.353936 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] base_mac = fa:16:3e:00:00:00 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.354108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] bind_host = 0.0.0.0 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.354200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] bind_port = 9696 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.354268 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] client_socket_timeout = 900 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.354363 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] config_dir = [] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.354422 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] config_file = ['/etc/neutron/neutron.conf', '/etc/neutron/plugins/ml2/ml2_conf.ini'] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.355590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] config_source = [] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.355753 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] control_exchange = neutron {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.355905 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] core_plugin = ml2 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.356051 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] debug = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.356221 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] default_availability_zones = [] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.356525 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] default_log_levels = ['amqp=WARN', 'boto=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'OFPHandler=INFO', 'OfctlService=INFO', 'os_ken.base.app_manager=INFO', 'os_ken.controller.controller=INFO'] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.356688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] dhcp_agent_notification = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.356835 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] dhcp_agents_per_network = 1 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.356968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] dhcp_lease_duration = 86400 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.357129 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] dhcp_load_type = networks {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.357272 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] dns_domain = openstacklocal {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.357424 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] dvr_base_mac = fa:16:3f:00:00:00 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.357574 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] enable_default_route_bfd = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.357700 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] enable_default_route_ecmp = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.357840 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] enable_dvr = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.357978 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] enable_new_agents = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.358112 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] enable_services_on_agents_with_admin_state_down = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.358266 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] enable_snat_by_default = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.358401 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] enable_traditional_dhcp = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.358533 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] external_dns_driver = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.358643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] fatal_deprecations = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.358766 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] filter_validation = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.358904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] global_physnet_mtu = 1430 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.359026 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] graceful_shutdown_timeout = 60 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.359313 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] host = np0000001841 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.359455 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] host_dvr_for_dhcp = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.359610 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] http_retries = 3 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.359740 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] instance_format = [instance: %(uuid)s] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.359843 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.359967 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ipam_driver = internal {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.360072 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ipv6_pd_enabled = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.360195 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] l3_ha = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.360328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_net_cidr = 169.254.192.0/18 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.360447 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_network_physical_name = {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.360573 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] l3_ha_network_type = {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.360679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] log_color = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.360799 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] log_config_append = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.360935 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.361051 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] log_dir = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.361156 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] log_file = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.361242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] log_options = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.361348 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] log_rotate_interval = 1 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.361461 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] log_rotate_interval_type = days {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.361586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] log_rotation_type = none {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.361686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.361765 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.361841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.361954 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.362035 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.362140 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] max_allowed_address_pair = 10 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.362253 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] max_dns_nameservers = 5 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.362356 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] max_header_line = 16384 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.362470 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] max_l3_agents_per_router = 3 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.362584 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] max_logfile_count = 30 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.362680 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] max_logfile_size_mb = 200 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.362781 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] max_routes = 30 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.362874 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] max_subnet_host_routes = 20 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.362990 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] my_ip = 10.4.3.71 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.363100 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] my_ipv6 = ::1 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.363223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] network_auto_schedule = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.363326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] network_link_prefix = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.363481 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] network_scheduler_driver = neutron.scheduler.dhcp_agent_scheduler.WeightScheduler {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.363618 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] notify_nova_on_port_data_changes = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.363748 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] notify_nova_on_port_status_changes = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.363865 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] pagination_max_limit = -1 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.363980 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] periodic_fuzzy_delay = 5 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.364077 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] periodic_interval = 40 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.364184 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] publish_errors = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.364285 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_burst = 0 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.364394 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_except_level = CRITICAL {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.364507 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] rate_limit_interval = 0 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.364631 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] router_distributed = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.364735 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] rpc_resources_processing_step = 20 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.364855 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] rpc_response_max_timeout = 600 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.364978 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] rpc_response_timeout = 60 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.365096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] rpc_state_report_workers = 0 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.365228 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] rpc_workers = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.365326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] send_events_interval = 2 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.365466 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] service_plugins = ['ovn-router'] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.365592 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] setproctitle = on {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.365696 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] shell_completion = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.365803 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] state_path = /opt/stack/data/neutron {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.365904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] syslog_log_facility = LOG_USER {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.366013 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] tcp_keepidle = 600 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.366133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] transport_url = **** {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.366259 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] use_journal = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.366386 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] use_json = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.366507 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] use_stderr = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.366716 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] use_syslog = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.366818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] vlan_qinq = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.366911 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] vlan_transparent = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.367004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] wsgi_default_pool_size = 100 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.367098 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] wsgi_keep_alive = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.367217 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] wsgi_log_format = %(client_ip)s "%(request_line)s" status: %(status_code)s len: %(body_length)s time: %(wall_seconds).7f {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.367348 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] wsgi_server_debug = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3245}} Jun 01 11:23:06.367530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_concurrency.disable_process_locking = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.367673 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_concurrency.lock_path = /opt/stack/data/neutron/lock {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.367801 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] profiler.connection_string = messaging:// {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.367922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] profiler.enabled = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.368024 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_doc_type = notification {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.368127 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_scroll_size = 10000 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.368242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] profiler.es_scroll_time = 2m {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.368351 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] profiler.filter_error_trace = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.368474 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] profiler.hmac_keys = **** {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.368604 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] profiler.sentinel_service_name = mymaster {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.368724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] profiler.socket_timeout = 0.1 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.368833 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] profiler.trace_requests = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.368930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] profiler.trace_sqlalchemy = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.369055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] profiler_jaeger.process_tags = {} {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.369151 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] profiler_jaeger.service_name_prefix = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.369261 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] profiler_otlp.service_name_prefix = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.369385 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.369492 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.369617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_process_name = {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.369736 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.369841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.369974 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.enforce_new_defaults = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.370063 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.enforce_scope = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.370147 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_default_rule = default {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.370247 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.370338 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.policy_file = /etc/neutron/policy.json {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.370449 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.370548 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.370636 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.370709 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.370783 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.370860 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_policy.remote_timeout = 60.0 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.370964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.371057 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] service_providers.service_provider = [] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.371148 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_section = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.371242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_type = password {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.371317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.auth_url = https://10.4.3.71/identity {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.371397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.cafile = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.371482 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.certfile = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.371578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.collect_timing = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.371653 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.default_domain_id = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.371725 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.default_domain_name = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.371798 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.domain_id = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.371871 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.domain_name = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.371944 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.endpoint_type = public {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.372017 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.insecure = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.372098 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.keyfile = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.372171 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.password = **** {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.372257 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.project_domain_id = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.372331 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.project_domain_name = Default {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.372402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.project_id = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.372474 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.project_name = service {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.372568 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.region_name = RegionOne {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.372649 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.split_loggers = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.372721 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.system_scope = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.372791 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.timeout = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.372862 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.trust_id = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.372959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.user_domain_id = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.373050 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.user_domain_name = Default {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.373134 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.user_id = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.373219 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] nova.username = nova {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.373315 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] placement.auth_section = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.373398 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] placement.auth_type = password {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.373470 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] placement.cafile = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.373548 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] placement.certfile = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.373633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] placement.collect_timing = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.373704 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] placement.endpoint_type = public {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.373775 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] placement.insecure = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.373847 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] placement.keyfile = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.373919 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] placement.region_name = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.373989 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] placement.split_loggers = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.374058 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] placement.timeout = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.374148 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.auth_section = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.374228 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.auth_type = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.374302 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.cafile = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.374372 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.certfile = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.374445 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.collect_timing = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.374517 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.connect_retries = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.374602 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.connect_retry_delay = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.374675 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.enable_notifications = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.374746 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.endpoint_override = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.374816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.insecure = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.374885 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.interface = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.374958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.keyfile = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.375027 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.max_version = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.375099 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.min_version = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.375169 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.region_name = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.375274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.retriable_status_codes = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.375391 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.service_name = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.375475 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.service_type = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.375568 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.split_loggers = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.375647 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.status_code_retries = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.375749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.status_code_retry_delay = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.375825 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.timeout = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.375897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.valid_interfaces = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.375971 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ironic.version = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.376057 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] cli_script.dry_run = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.376163 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.amqp_auto_delete = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.376251 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.amqp_durable_queues = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.376334 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.conn_pool_min_size = 2 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.376424 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.conn_pool_ttl = 1200 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.376494 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.direct_mandatory_flag = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.376586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.enable_cancel_on_failover = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.376669 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_rate = 3 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.376762 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.heartbeat_timeout_threshold = 60 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.376907 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.hostname = np0000001841 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.376996 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_compression = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.377069 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_failover_strategy = round-robin {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.377141 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_missing_consumer_retry_timeout = 60 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.377225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_reconnect_delay = 1.0 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.377312 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.kombu_reconnect_splay = 0.0 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.377384 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.processname = neutron-ovn-maintenance-worker {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.377458 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_ha_queues = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.377531 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_interval_max = 30 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.377618 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_login_method = AMQPLAIN {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.377723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_qos_prefetch_count = 0 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.377816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_delivery_limit = 0 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.377890 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_max_memory_bytes = 0 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.377963 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_max_memory_length = 0 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.378047 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_quorum_queue = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.378120 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_retry_backoff = 2 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.378201 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_retry_interval = 1 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.378274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_stream_fanout = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.378345 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_transient_queues_ttl = 1800 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.378431 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rabbit_transient_quorum_queue = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.378507 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.rpc_conn_pool_size = 30 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.378605 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.378681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_ca_file = {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.378753 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_cert_file = {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.378824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_enforce_fips_mode = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.378894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_enforce_hostname_verification = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.378965 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_key_file = {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.379035 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.ssl_version = {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.379112 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_rabbit.use_queue_manager = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.379223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.driver = [] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.379297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.retry = -1 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.379395 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.topics = ['notifications'] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.379499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_messaging_notifications.transport_url = **** {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.379608 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.file_event_handler = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.379683 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.file_event_handler_interval = 1 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.379759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_reports.log_dir = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.379849 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] oslo_versionedobjects.fatal_exception_format_errors = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.379948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.default_quota = -1 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.380034 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_driver = neutron.db.quota.driver_nolock.DbQuotaNoLockDriver {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.380125 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_floatingip = 50 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.380211 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_network = 100 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.380300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_port = 500 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.380377 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_rbac_policy = 10 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.380450 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_router = 10 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.380532 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_security_group = 10 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.380619 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_security_group_rule = 100 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.380692 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.quota_subnet = 100 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.380798 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] QUOTAS.track_quota_usage = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.380914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep.capabilities = [21, 12, 1, 2, 19] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.380996 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep.group = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.381069 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep.helper_command = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.381141 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep.log_daemon_traceback = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.381229 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep.logger_name = oslo_privsep.daemon {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.381310 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep.thread_pool_size = 12 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.381395 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep.user = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.381485 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.capabilities = [21, 12] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.381573 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.group = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.381649 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.helper_command = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.381725 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.log_daemon_traceback = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.381806 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.logger_name = oslo_privsep.daemon {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.381877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.thread_pool_size = 12 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.381948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_dhcp_release.user = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.382023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.capabilities = [21, 12] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.382095 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.group = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.382165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.helper_command = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.382250 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.log_daemon_traceback = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.382324 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.logger_name = oslo_privsep.daemon {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.382395 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.thread_pool_size = 12 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.382466 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_ovs_vsctl.user = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.382566 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.capabilities = [21] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.382656 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.group = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.382750 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.helper_command = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.382824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.log_daemon_traceback = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.382894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.logger_name = oslo_privsep.daemon {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.382964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.thread_pool_size = 12 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.383034 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_namespace.user = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.383122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.capabilities = [12] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.383203 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.group = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.383274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.helper_command = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.383344 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.log_daemon_traceback = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.383419 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.logger_name = oslo_privsep.daemon {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.383484 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.thread_pool_size = 12 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.383567 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_conntrack.user = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.383655 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.capabilities = [12, 21] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.383745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.group = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.383826 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.helper_command = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.383896 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.log_daemon_traceback = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.383968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.logger_name = oslo_privsep.daemon {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.384039 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.thread_pool_size = 12 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.384110 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] privsep_link.user = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.384222 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] AGENT.root_helper = sudo /opt/stack/data/venv/bin/neutron-rootwrap /etc/neutron/rootwrap.conf {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.384295 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] AGENT.root_helper_daemon = sudo /opt/stack/data/venv/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.384365 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] AGENT.use_helper_for_ns_read = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.384457 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.enable_ipset = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.384544 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.enable_security_group = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.384626 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.firewall_driver = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.384706 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] SECURITYGROUP.permitted_ethertypes = [] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.384796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] experimental.ipv6_pd_enabled = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.384898 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ml2.extension_drivers = ['port_security', 'qos'] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.384974 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ml2.external_network_type = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.385043 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ml2.mechanism_drivers = ['ovn'] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.385118 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ml2.overlay_ip_version = 4 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.385196 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ml2.path_mtu = 0 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.385272 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ml2.physical_network_mtus = {} {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.385359 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ml2.project_network_types = ['geneve'] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.385468 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ml2.tunnelled_network_rp_name = rp_tunnelled {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.385586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ml2.type_drivers = ['local', 'flat', 'vlan', 'geneve'] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.385692 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_flat.flat_networks = ['public'] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.385779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_geneve.max_header_size = 38 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.385876 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_geneve.vni_ranges = ['1:65536'] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.385995 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ml2_type_vlan.network_vlan_ranges = ['public'] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.386093 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] network_log.burst_limit = 25 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.386166 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] network_log.local_output_log_base = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.386250 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] network_log.rate_limit = 100 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.386345 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.asyncio_connection = **** {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.386430 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.asyncio_slave_connection = **** {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.386513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.backend = sqlalchemy {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.386613 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.connection = **** {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.386695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.connection_debug = 0 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.386767 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.connection_parameters = {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.386836 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.connection_recycle_time = 3600 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.386995 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.connection_trace = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.387078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.db_inc_retry_interval = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.387152 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.db_max_retries = 20 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.387249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.db_max_retry_interval = 10 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.387307 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.db_retry_interval = 1 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.387379 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.max_overflow = 50 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.387452 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.max_pool_size = 5 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.387524 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.max_retries = 10 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.387612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.mysql_sql_mode = TRADITIONAL {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.387685 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.mysql_wsrep_sync_wait = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.387760 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.pool_timeout = None {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.387832 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.retry_interval = 10 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.387916 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.slave_connection = **** {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.sqlite_synchronous = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.synchronous_reader = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388216 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] database.use_db_reconnect = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388318 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_min_rx = 1000 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388390 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_min_tx = 100 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388444 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.bfd_mult = 3 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.broadcast_arps_to_all_routers = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388522 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.dhcp_default_lease_time = 43200 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388576 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.disable_ovn_dhcp_for_baremetal_ports = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388623 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.dns_records_ovn_owned = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388669 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.dns_servers = [] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388710 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.enable_distributed_floating_ip = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.fdb_age_threshold = 300 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ha_failover_strategy = normal {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.live_migration_activation_strategy = rarp {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.localnet_learn_fdb = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388875 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.mac_binding_age_threshold = 300 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388910 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.neutron_sync_mode = log {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388940 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_dhcp4_global_options = {} {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.388972 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_dhcp6_global_options = {} {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389006 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_l3_scheduler = leastloaded {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389037 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_metadata_enabled = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389071 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389115 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389154 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_connection = ['ssl:10.4.3.71:6641'] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389202 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_nb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_router_indirect_snat = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_ca_cert = /opt/stack/data/CA/int-ca/ca-chain.pem {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389304 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_certificate = /opt/stack/data/CA/int-ca/devstack-cert.crt {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389338 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_connection = ['ssl:10.4.3.71:6642'] {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389372 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovn_sb_private_key = /opt/stack/data/CA/int-ca/private/devstack-cert.key {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389405 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovs_create_tap = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389447 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_connection_timeout = 180 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389447 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_log_level = INFO {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389515 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_probe_interval = 60000 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389515 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.ovsdb_retry_max_interval = 180 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.stateless_nat_enabled = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn.vhost_sock_dir = /var/run/openvswitch {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.389957 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] OVS.bridge_mac_table_size = 50000 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.390047 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.390124 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood_reports = True {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.390217 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_flood_unregistered = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.390301 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] OVS.igmp_snooping_enable = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.390397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] OVS.ovsdb_timeout = 10 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.390478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.fdb_removal_limit = 0 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.390574 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.ignore_lsp_down = False {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.390655 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ovn_nb_global.mac_binding_removal_limit = 0 {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3256}} Jun 01 11:23:06.390725 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** {{(pid=89427) log_opt_values /opt/stack/data/venv/lib/python3.12/site-packages/oslo_config/cfg.py:3263}} Jun 01 11:23:06.390915 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG cotyledon._service_worker [-] Run service ServiceWrapper(0) [89427] {{(pid=89427) wait_forever /opt/stack/data/venv/lib/python3.12/site-packages/cotyledon/_service_worker.py:218}} Jun 01 11:23:06.391821 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver.OVNMechanismDriver.post_fork_initialize-413670', 'neutron.services.ovn_l3.plugin.OVNL3RouterPlugin._post_fork_initialize-1619148'] for process (None), after_init {{(pid=89427) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Jun 01 11:23:06.392944 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Getting OvsdbNbOvnIdl for MaintenanceWorker with retry Jun 01 11:23:06.422422 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Logical_Switch.name {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 01 11:23:06.422516 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Logical_Router.name {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 01 11:23:06.422578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Load_Balancer.name {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 01 11:23:06.422660 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Copp.name {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 01 11:23:06.422728 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Meter.name {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 01 11:23:06.422792 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Load_Balancer_Group.name {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 01 11:23:06.422839 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index HA_Chassis_Group.name {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 01 11:23:06.422882 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Address_Set.name {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 01 11:23:06.422938 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Connection.target {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 01 11:23:06.422938 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Gateway_Chassis.name {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 01 11:23:06.423013 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Mirror.name {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 01 11:23:06.423013 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Chassis_Template_Var.chassis {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 01 11:23:06.423075 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Logical_Router_Port.name {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 01 11:23:06.423114 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Port_Group.name {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 01 11:23:06.423154 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Logical_Switch_Port.name {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 01 11:23:06.425357 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.4.3.71:6641: connecting... Jun 01 11:23:06.429239 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.4.3.71:6641: connected Jun 01 11:23:06.450802 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] Getting OvsdbSbOvnIdl for MaintenanceWorker with retry Jun 01 11:23:06.470902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Chassis.name {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 01 11:23:06.470989 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created lookup_table index Port_Binding.logical_port {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:87}} Jun 01 11:23:06.471040 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Chassis_Private.name {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 01 11:23:06.471040 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl [-] Created schema index Datapath_Binding.tunnel_key {{(pid=89427) autocreate_indices /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/__init__.py:106}} Jun 01 11:23:06.473011 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.4.3.71:6642: connecting... Jun 01 11:23:06.476976 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO ovsdbapp.backend.ovs_idl.vlog [-] ssl:10.4.3.71:6642: connected Jun 01 11:23:06.505902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip precommit_update 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.506007 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip precommit_delete 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.506094 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_create 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.506214 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > floatingip before_update 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.506307 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_update 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.506379 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port precommit_delete 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.506603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.services.portforwarding.drivers.ovn.driver.OVNPortForwarding.register-1618947'] for port_forwarding_plugin (None), after_init {{(pid=89427) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Jun 01 11:23:06.506702 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_create 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.506778 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > router_interface after_create 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.506850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_update 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.506921 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port_forwarding after_delete 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.506971 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > router_interface after_delete 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.507097 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension Jun 01 11:23:06.507193 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.placement [-] Starting OVNClientPlacementExtension Jun 01 11:23:06.507535 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment before_delete 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.507658 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet precommit_delete_associations 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.507951 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_create 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.508028 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment_host_mapping after_delete 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.508090 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_create 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.508148 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_delete 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.508225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_update 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.508308 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > segment after_delete 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.508375 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_create 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.508445 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.508503 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_update 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.575163 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_create 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.575296 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet after_delete 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.575296 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet before_create 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.575415 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > subnet before_update 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.578017 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group after_delete 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.578097 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > port after_delete 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.578232 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Publish callbacks ['neutron.services.logapi.drivers.ovn.driver.DriverBase._register-2914729'] for logging-plugin (None), after_init {{(pid=89427) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Jun 01 11:23:06.578354 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_create 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.578423 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_update 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:06.578498 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [-] Subscribe: > security_group_rule after_delete 55550000, False {{(pid=89427) subscribe /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:52}} Jun 01 11:23:16.099878 np0000001841 neutron-ovn-maintenance-worker[88598]: DEBUG dbcounter [-] [88598] Writing DB stats neutron:SELECT=4 {{(pid=88598) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:23:16.579451 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [-] OVN-Northbound DB sync process started @ 2026-06-01 11:23:16.579071 {{(pid=89427) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:99}} Jun 01 11:23:16.581206 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync port groups started @ 2026-06-01 11:23:16.580876 {{(pid=89427) sync_port_groups /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:132}} Jun 01 11:23:16.636960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writer thread running {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:102}} Jun 01 11:23:16.651649 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync port groups completed @ 2026-06-01 11:23:16.651375 {{(pid=89427) sync_port_groups /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:196}} Jun 01 11:23:16.651805 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync networks, ports and DHCP options started @ 2026-06-01 11:23:16.651647 {{(pid=89427) sync_networks_ports_and_dhcp_opts /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1152}} Jun 01 11:23:16.972610 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.services.segments.db [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] neutron.services.segments.plugin.Plugin method get_segments called with arguments (,) {'filters': {'network_id': ['c6c73189-2fc5-412b-9ed5-038efd7341bc'], 'is_dynamic': False}} {{(pid=89427) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} Jun 01 11:23:16.977464 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.services.segments.db [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] neutron.services.segments.plugin.Plugin method get_segments called with arguments (,) {'filters': {'network_id': ['0d4134e5-3b3a-49d4-a466-c1b6f993b0ab'], 'is_dynamic': False}} {{(pid=89427) wrapper /opt/stack/data/venv/lib/python3.12/site-packages/oslo_log/helpers.py:77}} Jun 01 11:23:16.982162 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync metadata ports started {{(pid=89427) _sync_metadata_ports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1097}} Jun 01 11:23:17.051820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync metadata ports completed {{(pid=89427) _sync_metadata_ports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1149}} Jun 01 11:23:17.051987 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync DHCP options for Neutron subnets started {{(pid=89427) _sync_subnet_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:949}} Jun 01 11:23:17.109501 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync DHCP options for Neutron subnets completed @ 2026-06-01 11:23:17.109290 {{(pid=89427) _sync_subnet_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1022}} Jun 01 11:23:17.109864 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 01 11:23:17.110061 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync DHCP options for Neutron ports with extra dhcp options assigned started {{(pid=89427) _sync_port_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1027}} Jun 01 11:23:17.110154 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync DHCP options for Neutron ports with extra dhcp options assigned completed {{(pid=89427) _sync_port_dhcp_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1086}} Jun 01 11:23:17.110284 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync networks, ports and DHCP options completed @ 2026-06-01 11:23:17.110149 {{(pid=89427) sync_networks_ports_and_dhcp_opts /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1321}} Jun 01 11:23:17.110386 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync ACLs started @ 2026-06-01 11:23:17.110299 {{(pid=89427) sync_acls /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:239}} Jun 01 11:23:17.143656 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync ACLs completed @ 2026-06-01 11:23:17.143398 {{(pid=89427) sync_acls /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:431}} Jun 01 11:23:17.143796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync routers and router ports started @ 2026-06-01 11:23:17.143654 {{(pid=89427) sync_routers_and_rports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:736}} Jun 01 11:23:17.369070 np0000001841 neutron-ovn-maintenance-worker[89427]: WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] Router Port found in Neutron but not in OVN NB DB, router port_id=92122edc-a57c-4880-996d-fb3c9c0822d7 Jun 01 11:23:17.370234 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 01 11:23:17.370497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync routers and router ports completed @ 2026-06-01 11:23:17.370309 {{(pid=89427) sync_routers_and_rports /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:944}} Jun 01 11:23:17.370661 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync port QoS policies started @ 2026-06-01 11:23:17.370507 {{(pid=89427) sync_port_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1369}} Jun 01 11:23:17.370791 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] Starting OVNClientQosExtension Jun 01 11:23:17.386828 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync port QoS policies completed @ 2026-06-01 11:23:17.386600 {{(pid=89427) sync_port_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1392}} Jun 01 11:23:17.386956 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync Floating IP QoS policies started @ 2026-06-01 11:23:17.386824 {{(pid=89427) sync_fip_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1397}} Jun 01 11:23:17.387057 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] Starting OVNClientQosExtension Jun 01 11:23:17.409839 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync Floating IP QoS policies completed @ 2026-06-01 11:23:17.409580 {{(pid=89427) sync_fip_qos_policies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1420}} Jun 01 11:23:17.410011 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync Floating IP NAT rules started @ 2026-06-01 11:23:17.409834 {{(pid=89427) sync_fip_dnat_rules /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1425}} Jun 01 11:23:17.410236 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-NB Sync Floating IP NAT rules completed @ 2026-06-01 11:23:17.410099 {{(pid=89427) sync_fip_dnat_rules /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1451}} Jun 01 11:23:17.410311 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-a78ce5eb-e9df-489d-98fd-b4aef86c2452 None None] OVN-Northbound DB sync process completed @ 2026-06-01 11:23:17.410223 {{(pid=89427) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:112}} Jun 01 11:23:26.580273 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [-] OVN-Southbound DB sync process started @ 2026-06-01 11:23:26.579955 {{(pid=89427) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1480}} Jun 01 11:23:26.580892 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.extensions.qos [-] Starting OVNClientQosExtension Jun 01 11:23:26.581714 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-1cf6488d-5f37-49bb-a0e8-2205e362eb56 None None] OVN-SB Sync hostname and physical networks started @ 2026-06-01 11:23:26.581482 {{(pid=89427) sync_hostname_and_physical_networks /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1511}} Jun 01 11:23:26.583400 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.583487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.583561 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_dhcp_options_consistency {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.583664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_fair_meter_consistency {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.583740 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_fdb_aging_settings {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.583817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_for_igmp_snoop_support {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.583880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_for_inconsistencies {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.583933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_global_dhcp_opts {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.583985 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.584036 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.584103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_provider_distributed_ports {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.586835 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.586884 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.586974 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.cleanup_old_hash_ring_nodes {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.588095 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-1cf6488d-5f37-49bb-a0e8-2205e362eb56 None None] Agents populated in the agent cache: {{(pid=89427) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:309}} Jun 01 11:23:26.588325 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.configure_nb_global {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.588498 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-1cf6488d-5f37-49bb-a0e8-2205e362eb56 None None] - Agent type: OVN Controller Gateway agent, hostname: np0000001841, name: f569aa1b-4c4f-4293-859b-ce5561ed7a02 {{(pid=89427) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:311}} Jun 01 11:23:26.588701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.588865 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.agent.neutron_agent [None req-1cf6488d-5f37-49bb-a0e8-2205e362eb56 None None] - Agent type: OVN Neutron agent, hostname: np0000001841, name: f569aa1b-4c4f-4293-859b-ce5561ed7a02 {{(pid=89427) populate /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py:311}} Jun 01 11:23:26.590121 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_fip_distributed_flag {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.590446 np0000001841 neutron-ovn-maintenance-worker[89427]: WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-1cf6488d-5f37-49bb-a0e8-2205e362eb56 None None] New host np0000001841 found in OVN SB DB, but not in Neutron. Add its SegmentHostMapping in Neutron Jun 01 11:23:26.590913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_network_type_and_physnet {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.590990 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.set_ovn_owned_dns_option {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.590990 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_ha_failover {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.591078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.591115 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_mac_aging_settings {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.591153 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_router_distributed_flag {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.591245 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_router_static_routes {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.591521 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_security_group_with_address_group {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.591679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [-] Periodic task found: DBInconsistenciesPeriodics.update_virtual_port_parent_hostname {{(pid=89427) add_periodics /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:148}} Jun 01 11:23:26.593388 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.597319 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [-] Maintenance task thread has started Jun 01 11:23:26.598292 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.598724 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [-] MaintenanceWorker process has finished the post initialization Jun 01 11:23:26.599900 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_dhcp_options_consistency' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.600045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fair_meter_consistency' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.600269 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.600349 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.600434 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.600499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_global_dhcp_opts' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.600607 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.600691 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.600754 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_provider_distributed_ports' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.600855 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.600944 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.601033 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.cleanup_old_hash_ring_nodes' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.601123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.configure_nb_global' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.601222 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.601314 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.601403 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_network_type_and_physnet' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.601499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_ovn_owned_dns_option' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.601596 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.601682 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.601768 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.601859 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_distributed_flag' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.602147 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_static_routes' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.602235 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_security_group_with_address_group' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.602309 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting immediate callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_virtual_port_parent_hostname' {{(pid=89427) _process_immediates /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:826}} Jun 01 11:23:26.606511 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG oslo_concurrency.lockutils [None req-1cf6488d-5f37-49bb-a0e8-2205e362eb56 None None] Acquiring lock "update_segment_host_mapping" by "neutron.services.segments.db.update_segment_host_mapping" {{(pid=89427) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:511}} Jun 01 11:23:26.606728 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG oslo_concurrency.lockutils [None req-1cf6488d-5f37-49bb-a0e8-2205e362eb56 None None] Lock "update_segment_host_mapping" acquired by "neutron.services.segments.db.update_segment_host_mapping" :: waited 0.000s {{(pid=89427) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:524}} Jun 01 11:23:26.618487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron_lib.callbacks.manager [None req-1cf6488d-5f37-49bb-a0e8-2205e362eb56 None None] Publish callbacks ['neutron.plugins.ml2.plugin.DhcpAgentSchedulerDbMixin.auto_schedule_new_network_segments-871688', 'neutron.services.segments.plugin.NovaSegmentNotifier._notify_host_addition_to_aggregate-3710498'] for segment_host_mapping (None), after_create {{(pid=89427) _notify_loop /opt/stack/data/venv/lib/python3.12/site-packages/neutron_lib/callbacks/manager.py:184}} Jun 01 11:23:26.630371 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.services.segments.db [None req-1cf6488d-5f37-49bb-a0e8-2205e362eb56 None None] Segments {'4ecb0f57-2a36-41bc-88d1-5cf4f555b097'} mapped to the host np0000001841 {{(pid=89427) update_segment_host_mapping /opt/stack/neutron/neutron/services/segments/db.py:249}} Jun 01 11:23:26.638622 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG oslo_concurrency.lockutils [None req-1cf6488d-5f37-49bb-a0e8-2205e362eb56 None None] Lock "update_segment_host_mapping" "released" by "neutron.services.segments.db.update_segment_host_mapping" :: held 0.032s {{(pid=89427) inner /opt/stack/data/venv/lib/python3.12/site-packages/oslo_concurrency/lockutils.py:543}} Jun 01 11:23:26.638760 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-1cf6488d-5f37-49bb-a0e8-2205e362eb56 None None] OVN-SB Sync hostname and physical networks completed @ 2026-06-01 11:23:26.638610 {{(pid=89427) sync_hostname_and_physical_networks /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1538}} Jun 01 11:23:26.658690 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.add_provnet_ext_id_to_localnet_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:26.658984 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_baremetal_ports_dhcp_options {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:26.659075 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Sync baremetal port DHCP options with configuration. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:26.712121 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_baremetal_ports_dhcp_options finished in 0.053 seconds Jun 01 11:23:26.712268 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_baremetal_ports_dhcp_options' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:26.712412 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_dhcp_options_consistency {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:26.715637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Backfill network_id on DHCP_Options external_ids. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:26.774424 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_dhcp_options_consistency finished in 0.059 seconds Jun 01 11:23:26.774655 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_dhcp_options_consistency' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:26.774655 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fair_meter_consistency {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:26.774742 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Refresh ACL logging fair meter after configuration reload. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:26.774989 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fair_meter_consistency finished in 0.000 seconds Jun 01 11:23:26.775086 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fair_meter_consistency' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:26.775191 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:26.775286 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:26.776459 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:27.030081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:27.030347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.032830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:27.032830 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.032830 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:27.032830 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:27.032830 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.032830 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:27.032830 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.032830 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:27.038185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.258 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.038185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:27.038185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:27.038185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:27.038185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.038185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:23:27.038185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:27.038185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.038185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:27.038185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:27.038185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:27.038185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.038493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:27.038493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:27.038493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:27.038493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:27.038493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:27.038493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:27.038493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:27.038493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:27.038493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.038493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:27.038493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:27.038493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.038493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:27.038806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.038806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:27.041737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.041737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:27.041737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:27.041737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:27.041737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:27.041737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:27.041737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.041737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:27.041737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:27.041737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.041737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:27.041737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:27.041737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.041737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:27.041737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:27.041737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:27.042144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.042510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:27.042510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.042510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:27.042510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_for_igmp_snoop_support {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.042510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Check for IGMP snoop support. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.042742 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_for_igmp_snoop_support finished in 0.000 seconds Jun 01 11:23:27.042875 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:27.043004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.043095 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.057777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: No inconsistencies found. Skipping {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:437}} Jun 01 11:23:27.058424 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_for_inconsistencies finished in 0.015 seconds Jun 01 11:23:27.058667 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_global_dhcp_opts {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.058781 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Check global DHCP options consistency. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.058987 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_global_dhcp_opts finished in 0.000 seconds Jun 01 11:23:27.059123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_global_dhcp_opts' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:27.059514 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_localnet_port_has_learn_fdb {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.059514 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Ensure localnet ports have learn-fdb set per configuration. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.060724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 01 11:23:27.060953 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_localnet_port_has_learn_fdb finished in 0.001 seconds Jun 01 11:23:27.061090 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_localnet_port_has_learn_fdb' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:27.061258 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_network_broadcast_arps_to_all_routers {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.061358 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Sync broadcast-arps-to-all-routers on external networks. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.104369 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_network_broadcast_arps_to_all_routers finished in 0.043 seconds Jun 01 11:23:27.104497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_network_broadcast_arps_to_all_routers' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:27.104652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_provider_distributed_ports {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.104720 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Align reside-on-redirect-chassis on VLAN/FLAT distributed ports. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.168024 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 01 11:23:27.168259 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [None req-1cf6488d-5f37-49bb-a0e8-2205e362eb56 None None] OVN-Southbound DB sync process completed @ 2026-06-01 11:23:27.168109 {{(pid=89427) do_sync /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_db_sync.py:1488}} Jun 01 11:23:27.177060 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_provider_distributed_ports finished in 0.072 seconds Jun 01 11:23:27.177166 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_provider_distributed_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:27.177343 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_redirect_type_router_gateway_ports {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.177433 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Align router gateway redirect-type for provider VLAN/FLAT networks. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.229377 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_redirect_type_router_gateway_ports finished in 0.052 seconds Jun 01 11:23:27.229527 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_redirect_type_router_gateway_ports' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:27.229718 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.229874 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.230703 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:27.230851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.231293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:27.231293 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.231293 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:27.231293 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:27.231293 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.231293 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:27.231293 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.231293 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:27.232137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.232137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:27.232137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:27.232137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:27.232137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.232137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:23:27.232137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:27.232137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.232137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:27.232137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:27.232137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:27.232137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.232496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:27.232496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:27.232496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:27.232496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:27.232496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:27.232496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:27.232496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:27.232496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:27.232496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.232496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:27.232496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:27.232496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.232496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:27.232871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.232871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:27.232871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.232871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:27.232871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:27.232871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:27.232871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:27.232871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:27.232871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.232871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:27.232871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:27.232871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.232871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:27.232871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:27.232871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:27.233307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.233838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:27.233838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:27.233838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.233838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:27.233838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.233838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:27.233838 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: cleanup_old_hash_ring_nodes {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.233838 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Purge stale hash ring nodes older than five days. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.236907 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.db.ovn_hash_ring_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Cleaned up Hash Ring nodes older than 5 days Jun 01 11:23:27.237013 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task cleanup_old_hash_ring_nodes finished in 0.004 seconds Jun 01 11:23:27.237157 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: configure_nb_global {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.237303 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply ovn_nb_global settings to NB_Global options. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.237479 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'ignore_lsp_down': 'false', 'fdb_removal_limit': '0', 'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:27.237758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'ignore_lsp_down': 'false', 'fdb_removal_limit': '0', 'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:27.237945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.238305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:27.238305 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.238305 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:27.238305 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:27.238305 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.238305 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:27.238305 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.238305 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:27.239364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.239364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:23:27.239364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:27.239364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:27.239364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.239364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:27.239364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:27.239364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:23:27.239364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.239364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:27.239364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:27.239364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:23:27.239364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:27.239364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:23:27.239364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:23:27.239867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:27.239867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:23:27.239867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.239867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:23:27.239867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:23:27.239867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.239867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:23:27.239867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.239867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:23:27.240223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task configure_nb_global failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.240223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:27.240223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:27.240223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:27.240223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.240223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1038, in configure_nb_global Jun 01 11:23:27.240223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:23:27.240223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:27.240223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:27.240223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.240223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:27.240564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:27.240564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:27.240564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.240564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:27.240564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:27.240564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:27.240564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:27.240564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:27.240564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:27.240564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:27.240564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:27.240564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.240564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:27.240564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:27.240950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.240950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:27.240950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.240950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:27.240950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.configure_nb_global' (it runs every 86400.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.240950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:27.240950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:27.240950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:27.240950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:27.240950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:27.240950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.240950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:27.240950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:27.240950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1038, in configure_nb_global Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:27.241356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:27.241884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:27.241884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:27.241884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:27.241884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:27.241884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.241884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:27.241884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:27.241884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.241884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:27.241884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.241884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:27.241884 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] There are no commands to commit {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:78}} Jun 01 11:23:27.241884 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.migrate_lrp_gateway_chassis_to_ha_chassis_group' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:27.242262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.242262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.242262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:23:27.242262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:27.242262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.242432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:27.242432 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.242432 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:27.242432 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:27.242432 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.242432 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:27.242432 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.242432 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:27.242905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.242905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:23:27.242905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:27.242905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:27.242905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.242905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:27.242905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:27.242905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:23:27.242905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.242905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:27.242905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:27.242905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:23:27.242905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:27.242905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:23:27.242905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:23:27.243329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:27.243329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:23:27.243329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.243329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:23:27.243329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:23:27.243329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.243329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:23:27.243329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.243329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:23:27.243662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.243662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:27.243662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:27.243662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:27.243662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.243662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:23:27.243662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:23:27.243662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:23:27.243662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:27.243662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:27.243662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.243662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:27.244001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:27.244001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:27.244001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.244001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:27.244001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:27.244001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:27.244001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:27.244001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:27.244001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:27.244001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:27.244001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:27.244001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.244001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:27.244001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:27.244408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.244408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:27.244408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.244408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:27.244408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.244408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:27.244408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:27.244408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:27.244408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:27.244408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:27.244408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.244408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:27.244408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:27.244408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:27.245182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:27.245992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:27.245992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:27.245992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:27.245992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:27.245992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:27.245992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.245992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:27.245992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:27.245992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.245992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:27.245992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.245992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:27.245992 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_network_type_and_physnet {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.245992 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set network type and physnet external_ids on logical switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.249817 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_network_type_and_physnet finished in 0.005 seconds Jun 01 11:23:27.249960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_network_type_and_physnet' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:27.250088 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_ovn_owned_dns_option {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.250221 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply ovn_owned option on DNS records per configuration. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.250400 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_ovn_owned_dns_option finished in 0.000 seconds Jun 01 11:23:27.250508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_ovn_owned_dns_option' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:27.250654 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.250733 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.250959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:27.251497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:27.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.252218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:27.252218 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.252218 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:27.252218 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:27.252218 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.252218 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:27.252218 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.252218 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:27.252784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.252784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:23:27.252784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:27.252784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:27.252784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.252784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:27.252784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:27.252784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:23:27.252784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.252784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:27.252784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:27.252784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:23:27.252784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:27.252784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:23:27.252784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:23:27.253325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:27.253325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:23:27.253325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.253325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:23:27.253325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:23:27.253325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.253325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:23:27.253325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.253325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:23:27.253564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.253564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:27.253564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:27.253564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:27.253564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.253564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:23:27.253564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:23:27.253564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:27.253564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:27.253564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.253564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:27.253870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:27.253870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:27.253870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.253870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:27.253870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:27.253870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:27.253870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:27.253870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:27.253870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:27.253870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:27.253870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:27.253870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.253870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:27.253870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:27.254277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.254277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:27.254277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.254277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:27.254277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.254277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:27.254277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:27.254277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:27.254277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:27.254277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:27.254277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.254277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:27.254277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:27.254277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:27.254705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:27.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:27.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:27.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:27.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:27.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:27.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:27.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:27.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:27.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_lrouter_ports_ext_ids_name_prefix {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: ('Check missing prefix router_name', 'in external_ids of LRPs.') {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.255668 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_lrouter_ports_ext_ids_name_prefix finished in 0.000 seconds Jun 01 11:23:27.255668 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_lrouter_ports_ext_ids_name_prefix' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:27.255668 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.255668 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.255668 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:27.257777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:27.257940 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:27.258009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.258390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:27.258390 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.258390 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:27.258390 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:27.258390 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.258390 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:27.258390 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.258390 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:27.258990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.258990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:27.258990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:27.258990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:27.258990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.258990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:23:27.258990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:27.258990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.258990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:27.258990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:27.258990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:27.258990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.259371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:27.259371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:27.259371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:27.259371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:27.259371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:27.259371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:27.259371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:27.259371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:27.259371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.259371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:27.259371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:27.259371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.259371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:27.259820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.259820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:27.259820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call immediate 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.259820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:27.259820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:27.259820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:27.259820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:27.259820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:27.259820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.259820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:27.259820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:27.259820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.259820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:27.259820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:27.259820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:27.260306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:27.260897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:27.260897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:27.260897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:27.260897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:27.260897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:27.260897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:27.260897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_router_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.260897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Sync router.distributed with distributed floating IP config. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.263001 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_router_distributed_flag finished in 0.003 seconds Jun 01 11:23:27.263101 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_distributed_flag' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:27.263230 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_router_static_routes {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.263346 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Mark Neutron-owned static routes in OVN external_ids. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.263984 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_router_static_routes finished in 0.001 seconds Jun 01 11:23:27.264077 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_router_static_routes' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:27.264194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_security_group_with_address_group {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:27.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Repair Address_Sets and ACLs for address group rules. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:27.273468 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_security_group_with_address_group finished in 0.009 seconds Jun 01 11:23:27.273531 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_security_group_with_address_group' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:27.274506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_virtual_port_parent_hostname' raised 'NeverAgain' exception, stopping any further execution of it. {{(pid=89427) _on_done /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:881}} Jun 01 11:23:31.775526 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:31.777957 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:31.777957 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:31.777957 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:31.801182 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:31.801393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:31.803190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:31.803190 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:31.803190 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:31.803190 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:31.803190 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:31.803190 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:31.803190 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:31.803190 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:31.803190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:31.803190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:31.803190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:31.803190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:31.803190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:31.803190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:23:31.803190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:31.803608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:31.803608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:31.803608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:31.803608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:31.803608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:31.803608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:31.803608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:31.803608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:31.803608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:31.803608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:31.803608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:31.803608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:31.803608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:31.803608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:31.803862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:31.803862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:31.803862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:31.803862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:31.803862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:31.803862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:31.803862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:31.803862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:31.803862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:31.803862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:31.803862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:31.803862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:31.803862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:31.803862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:31.804100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:31.804559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:31.804559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:31.804559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:31.804559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:31.804559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:31.804559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:31.804559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:31.804559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:31.804559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:32.231823 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:32.232649 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:32.232649 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:32.233922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:32.234117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.236482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:32.236482 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:32.236482 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:32.236482 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:32.236482 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:32.236482 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:32.236482 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.236482 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:32.236482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.236482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:32.236482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:32.236482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:32.236482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.236482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:23:32.236482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:32.236915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.236915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:32.236915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:32.236915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:32.236915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.236915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:32.236915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:32.236915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:32.236915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:32.236915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:32.236915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:32.236915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:32.236915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:32.236915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:32.237190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:32.237190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:32.237190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:32.237190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:32.237190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.237190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:32.237190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.237190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:32.237190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:32.237190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:32.237190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:32.237190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:32.237190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.237190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:32.237475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:32.237999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:32.237999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:32.237999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:32.237999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:32.237999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:32.237999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:32.237999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:32.237999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:32.237999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.237999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:32.241586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:32.242138 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:32.242138 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:32.242138 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:23:32.242491 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:32.242749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.245191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:32.245621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:32.245621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:23:32.245621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:32.245621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:23:32.245621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:23:32.245621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:32.245621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:23:32.245621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:32.245621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:23:32.245621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:23:32.245621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:32.245621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:23:32.245621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.245621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:23:32.245621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.245918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:32.245918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:32.245918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:32.245918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.245918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:23:32.245918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:23:32.245918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:23:32.245918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:32.245918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:32.245918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.245918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:32.245918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:32.245918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:32.246162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.246162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:32.246162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:32.246162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:32.246162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:32.246162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:32.246162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:32.246162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:32.246162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:32.246162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:32.246162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:32.246162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:32.246162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:32.246162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:32.246414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.246414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:32.246414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.246414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:32.246414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:32.246414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:32.246414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:32.246414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:32.246414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.246414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:32.246414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:32.246414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.246414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:32.246414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:32.246414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:32.246722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:32.247381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:32.247381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:32.247381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:32.247381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:32.247381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:32.247381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:32.247381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:32.247381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.247381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:32.250708 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:32.251239 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:32.251239 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:32.251239 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:32.254029 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:32.254321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:32.256287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:32.256287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:23:32.256287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:32.256287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:23:32.256287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:23:32.256287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:32.256287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:23:32.256287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:32.256287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:23:32.256287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:23:32.256287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:32.256287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:23:32.256287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.256287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:23:32.257115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.257115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:32.257115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:32.257115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:32.257115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.257115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:23:32.257115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:23:32.257115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:32.257115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:32.257115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.257115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:32.257931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:32.257931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:32.257931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.257931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:32.257931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:32.257931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:32.257931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:32.257931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:32.257931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:32.257931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:32.257931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:32.257931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:32.257931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:32.257931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:32.260159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:32.260159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:32.260159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.260159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:32.260159 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:32.260159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.260159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:32.260159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:32.260159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:32.260159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:32.260159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:32.260159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.260815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:32.261410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:32.261410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:32.261410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:32.261410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:32.261410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:32.261410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:32.261410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:32.261410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:32.261410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:32.261410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:32.261410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:32.261410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:32.261410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.261410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:32.261410 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:32.261743 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:32.261743 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:32.262406 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:32.262621 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:32.262721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.264955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:32.264955 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:32.264955 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:32.264955 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:32.264955 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:32.264955 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:32.264955 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.264955 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:32.264955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.264955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:32.264955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:32.264955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:32.264955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.264955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:23:32.264955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:32.265351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.265351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:32.265351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:32.265351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:32.265351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.265351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:32.265351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:32.265351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:32.265351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:32.265351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:32.265351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:32.265351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:32.265351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:32.265351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:32.265643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:32.265643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:32.265643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:32.265643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:32.265643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.265643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:32.265643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.265643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:32.265643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:32.265643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:32.265643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:32.265643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:32.265643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.265643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:32.265909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:32.266290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:32.266290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:32.266290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:32.266290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:32.266290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:32.266290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:32.266290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:32.266290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:32.266290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:32.266290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:36.776430 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:36.776933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:36.776933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:36.777559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:36.805937 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:36.806106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:36.806681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:36.806681 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:36.806681 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:36.806681 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:36.806681 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:36.806681 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:36.806681 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:36.806681 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:36.807527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:36.807527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:36.807527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:36.807527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:36.807527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:36.807527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:23:36.807527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:36.807527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:36.807527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:36.807527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:36.807527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:36.807527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:36.807792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:36.807792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:36.807792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:36.807792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:36.807792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:36.807792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:36.807792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:36.807792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:36.807792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:36.807792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:36.807792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:36.807792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:36.807792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:36.808036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:36.808036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:36.808250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:36.808250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:36.808250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:36.808250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:36.808250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:36.808250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:36.808250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:36.808250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:36.808250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:36.808250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:36.808250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:36.808250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:36.808250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:36.808250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:36.808250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:36.808250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:36.808551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:36.808914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:36.808914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:36.808914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:37.232319 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:37.232690 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:37.232827 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:37.233246 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:37.233388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.233824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:37.233824 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:37.233824 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:37.233824 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:37.233824 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:37.233824 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:37.233824 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.233824 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:37.234441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.234441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:37.234441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:37.234441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:37.234441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.234441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:23:37.234441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:37.234441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.234441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:37.234441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:37.234441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:37.234441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.234673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:37.234673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:37.234673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:37.234673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:37.234673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:37.234673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:37.234673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:37.234673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:37.234673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:37.234673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:37.234673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:37.234673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:37.234673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:37.234888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.234888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:37.235135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.235135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:37.235135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:37.235135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:37.235135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:37.235135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:37.235135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.235135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:37.235135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:37.235135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.235135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:37.235135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:37.235135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.235135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:37.235135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:37.235135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:37.235412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:37.235739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:37.235739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.235739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:37.241943 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:37.242154 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:37.242249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:37.242337 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:23:37.242568 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:37.242716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.243030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:37.243030 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:37.243030 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:37.243030 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:37.243030 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:37.243030 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:37.243030 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.243030 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:37.243513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.243513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:23:37.243513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:37.243513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:37.243513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.243513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:37.243513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:37.243513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:23:37.243513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.243513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:37.243513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:37.243513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:23:37.243513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:37.243513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:23:37.243513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:23:37.243776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:37.243776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:23:37.243776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:37.243776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:23:37.243776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:23:37.243776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:37.243776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:23:37.243776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.243776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:23:37.244078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.244078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:37.244078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:37.244078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:37.244078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.244078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:23:37.244078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:23:37.244078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:23:37.244078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:37.244078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:37.244078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.244078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:37.244344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:37.244344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:37.244344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.244344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:37.244344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:37.244344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:37.244344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:37.244344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:37.244344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:37.244344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:37.244344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:37.244344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:37.244344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:37.244344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:37.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:37.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:37.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:37.244761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.244761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:37.244761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:37.244761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:37.244761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:37.244761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:37.244761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.244761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:37.244761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:37.244761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.244761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:37.244761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:37.244761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.244761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:37.244761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:37.244761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:37.245023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:37.245318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:37.245318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:37.245318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:37.245318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:37.245318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:37.245318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.245318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:37.251019 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:37.251188 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:37.251300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:37.251394 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:37.251715 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:37.251895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.252221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:37.252221 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:37.252221 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:37.252221 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:37.252221 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:37.252221 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:37.252221 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.252221 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:37.252808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.252808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:23:37.252808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:37.252808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:37.252808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.252808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:37.252808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:37.252808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:23:37.252808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.252808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:37.252808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:37.252808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:23:37.252808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:37.252808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:23:37.252808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:23:37.253053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:37.253053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:23:37.253053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:37.253053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:23:37.253053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:23:37.253053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:37.253053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:23:37.253053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.253053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:23:37.253451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.253451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:37.253451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:37.253451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:37.253451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.253451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:23:37.253451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:23:37.253451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:37.253451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:37.253451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.253451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:37.253647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:37.253647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:37.253647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.253647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:37.253647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:37.253647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:37.253647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:37.253647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:37.253647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:37.253647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:37.253647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:37.253647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:37.253647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:37.253647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:37.253869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:37.253869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:37.253869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.253869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.254154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:37.254447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:37.254741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:37.254741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:37.254741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.254741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:37.259958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:37.260139 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:37.260236 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:37.260385 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:37.260697 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:37.260904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:37.261005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.261451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:37.261451 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:37.261451 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:37.261451 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:37.261451 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:37.261451 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:37.261451 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.261451 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:37.262162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.262162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:37.262162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:37.262162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:37.262162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.262162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:23:37.262162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:37.262162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.262162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:37.262162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:37.262162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:37.262162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.262398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:37.262398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:37.262398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:37.262398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:37.262398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:37.262398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:37.262398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:37.262398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:37.262398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:37.262398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:37.262398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:37.262398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:37.262398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:37.262635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.262635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:37.262850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.262850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:37.262850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:37.262850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:37.262850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:37.262850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:37.262850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.262850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:37.262850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:37.262850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.262850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:37.262850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:37.262850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.262850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:37.262850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:37.262850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:37.263107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:37.263417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:37.263417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:37.263417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:41.776850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:41.777633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:41.777633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:41.777989 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:41.805960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:41.808593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:41.808593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:41.808593 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:41.808593 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:41.808593 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:41.808593 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:41.808593 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:41.808593 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:41.808593 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:41.808593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:41.808593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:41.808593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:41.808593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:41.808593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:41.808593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:23:41.808973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:41.808973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:41.808973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:41.808973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:41.808973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:41.808973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:41.808973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:41.808973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:41.808973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:41.808973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:41.808973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:41.808973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:41.808973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:41.808973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:41.809293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:41.809293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:41.809293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:41.809293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:41.809293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:41.809293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:41.809293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:41.809293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:41.809293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:41.809293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:41.809293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:41.809293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:41.809293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:41.809599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:41.809980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:41.809980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:41.809980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:41.809980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:41.809980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:41.809980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:41.809980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:41.809980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:41.809980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:41.809980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:41.809980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:41.809980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:42.232776 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:42.233116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:42.233284 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:42.234003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:42.234201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.234688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:42.234688 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:42.234688 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:42.234688 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:42.234688 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:42.234688 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:42.234688 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.234688 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:42.235379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.235379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:42.235379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:42.235379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:42.235379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.235379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:23:42.235379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:42.235379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.235379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:42.235379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:42.235379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:42.235379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.235739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:42.235739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:42.235739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:42.235739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:42.235739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:42.235739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:42.235739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:42.235739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:42.235739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:42.235739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:42.235739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:42.235739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:42.235739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:42.236053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.236053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:42.236114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.236114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:42.236114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:42.236114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:42.236114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:42.236114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:42.236114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.236114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:42.236114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:42.236114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.236114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:42.236114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:42.236114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.236114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:42.236114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:42.236114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:42.236510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:42.236993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:42.236993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.236993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:42.242391 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:42.242627 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:42.242734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:42.242813 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:23:42.243228 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:42.243428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.243858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:42.243858 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:42.243858 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:42.243858 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:42.243858 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:42.243858 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:42.243858 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.243858 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:42.244429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.244429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:23:42.244429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:42.244429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:42.244429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.244429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:42.244429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:42.244429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:23:42.244429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.244429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:42.244429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:42.244429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:23:42.244429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:42.244429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:23:42.244429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:23:42.244710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:42.244710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:23:42.244710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:42.244710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:23:42.244710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:23:42.244710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:42.244710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:23:42.244710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.244710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:23:42.245047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.245047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:42.245047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:42.245047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:42.245047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.245047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:23:42.245047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:23:42.245047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:23:42.245047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:42.245047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:42.245047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.245047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:42.245654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:42.245654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:42.245654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.245654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:42.245654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:42.245654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:42.245654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:42.245654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:42.245654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:42.245654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:42.245654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:42.245654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:42.245654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:42.245654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:42.246006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:42.246006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:42.246006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.246006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:42.246006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.246006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:42.246006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:42.246006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:42.246006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:42.246006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:42.246006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.246006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:42.246006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:42.246006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:42.246240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:42.246532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:42.246532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:42.246532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:42.246532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:42.246532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:42.246532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:42.246532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:42.246532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:42.246532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:42.246532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:42.246532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.246532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:42.251440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:42.251714 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:42.251811 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:42.251917 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:42.252390 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:42.252649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.254995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:42.254995 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:42.254995 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:42.254995 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:42.254995 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:42.254995 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:42.254995 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.254995 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:42.255646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.255646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:23:42.255646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:42.255646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:42.255646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.255646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:42.255646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:42.255646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:23:42.255646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.255646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:42.255646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:42.255646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:23:42.255646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:42.255646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:23:42.255646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:23:42.255901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:42.255901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:23:42.255901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:42.255901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:23:42.255901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:23:42.255901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:42.255901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:23:42.255901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.255901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:23:42.256345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.256345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:42.256345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:42.256345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:42.256345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.256345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:23:42.256345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:23:42.256345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:42.256345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:42.256345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.256345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:42.256557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:42.256557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:42.256557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.256557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:42.256557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:42.256557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:42.256557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:42.256557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:42.256557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:42.256557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:42.256557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:42.256557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:42.256557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:42.256557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:42.256772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:42.256772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:42.256772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.256772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:42.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:42.258950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:42.258950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:42.258950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.258950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:42.260194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:42.260392 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:42.260479 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:42.260651 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:42.261154 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:42.261490 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:42.261663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.262256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:42.262256 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:42.262256 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:42.262256 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:42.262256 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:42.262256 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:42.262256 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.262256 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:42.262837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.262837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:42.262837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:42.262837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:42.262837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.262837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:23:42.262837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:42.262837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.262837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:42.262837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:42.262837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:42.262837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.263087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:42.263087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:42.263087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:42.263087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:42.263087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:42.263087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:42.263087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:42.263087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:42.263087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:42.263087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:42.263087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:42.263087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:42.263087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:42.263390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.263390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:42.263722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.263722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:42.263722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:42.263722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:42.263722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:42.263722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:42.263722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.263722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:42.263722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:42.263722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.263722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:42.263722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:42.263722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.263722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:42.263722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:42.263722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:42.264481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:42.264875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:42.264875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:42.264875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:46.777547 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:46.778120 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:46.778120 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:46.778567 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:46.820979 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:46.821157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:46.821620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:46.821620 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:46.821620 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:46.821620 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:46.821620 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:46.821620 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:46.821620 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:46.821620 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:46.822635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:46.822635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:46.822635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:46.822635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:46.822635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:46.822635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:23:46.822635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:46.822635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:46.822635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:46.822635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:46.822635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:46.822635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:46.822912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:46.822912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:46.822912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:46.822912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:46.822912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:46.822912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:46.822912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:46.822912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:46.822912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:46.822912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:46.822912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:46.822912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:46.822912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:46.823235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:46.823235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:46.823386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:46.823386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:46.823386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:46.823386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:46.823386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:46.823386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:46.823386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:46.823386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:46.823386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:46.823386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:46.823386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:46.823386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:46.823386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:46.823386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:46.823386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:46.823386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:46.823790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:46.824303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:46.824303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:46.824303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:47.233146 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:47.233415 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:47.233525 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:47.233989 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:47.234176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.234710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:47.234710 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:47.234710 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:47.234710 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:47.234710 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:47.234710 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:47.234710 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.234710 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:47.235377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.235377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:47.235377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:47.235377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:47.235377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.235377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:23:47.235377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:47.235377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.235377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:47.235377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:47.235377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:47.235377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.235710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:47.235710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:47.235710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:47.235710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:47.235710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:47.235710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:47.235710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:47.235710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:47.235710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:47.235710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:47.235710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:47.235710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:47.235710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:47.236054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.236054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:47.236119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.236119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:47.236119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:47.236119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:47.236119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:47.236119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:47.236119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.236119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:47.236119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:47.236119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.236119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:47.236119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:47.236119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.236119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:47.236119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:47.236119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:47.236517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:47.237005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:47.237005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.237005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:47.242730 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:47.243006 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:47.243123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:47.243235 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:23:47.243834 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:47.244032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.245598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:47.245598 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:47.245598 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:47.245598 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:47.245598 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:47.245598 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:47.245598 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.245598 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:47.246215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.246215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:23:47.246215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:47.246215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:47.246215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.246215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:47.246215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:47.246215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:23:47.246215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.246215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:47.246215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:47.246215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:23:47.246215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:47.246215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:23:47.246215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:23:47.246551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:47.246551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:23:47.246551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:47.246551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:23:47.246551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:23:47.246551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:47.246551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:23:47.246551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.246551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:23:47.246900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.246900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:47.246900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:47.246900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:47.246900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.246900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:23:47.246900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:23:47.246900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:23:47.246900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:47.246900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:47.246900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.246900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:47.247127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:47.247127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:47.247127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.247127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:47.247127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:47.247127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:47.247127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:47.247127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:47.247127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:47.247127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:47.247127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:47.247127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:47.247127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:47.247127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:47.247394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:47.247394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:47.247394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.247394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:47.247655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.247655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:47.247655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:47.247655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:47.247655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:47.247655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:47.247655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.247655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:47.247655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:47.247655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.247655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:47.247655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:47.247655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.247655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:47.247655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:47.247655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:47.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:47.248273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:47.248273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:47.248273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:47.248273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:47.248273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:47.248273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.248273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:47.251734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:47.251923 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:47.252008 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:47.252107 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:47.252521 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:47.253351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.253831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:47.253831 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:47.253831 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:47.253831 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:47.253831 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:47.253831 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:47.253831 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.253831 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:47.254614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.254614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:23:47.254614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:47.254614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:47.254614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.254614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:47.254614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:47.254614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:23:47.254614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.254614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:47.254614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:47.254614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:23:47.254614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:47.254614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:23:47.254614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:23:47.254964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:47.254964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:23:47.254964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:47.254964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:23:47.254964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:23:47.254964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:47.254964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:23:47.254964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.254964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:23:47.255295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.255295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:47.255295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:47.255295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:47.255295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.255295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:23:47.255295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:23:47.255295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:47.255295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:47.255295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.255295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:47.255513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:47.255513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:47.255513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.255513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:47.255513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:47.255513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:47.255513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:47.255513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:47.255513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:47.255513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:47.255513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:47.255513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:47.255513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:47.255513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:47.255772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:47.255772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:47.255772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.255772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:47.256467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:47.256899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:47.256899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:47.256899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.256899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:47.260453 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:47.260655 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:47.260744 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:47.260898 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:47.263325 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:47.263516 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:47.263633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.264829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:47.264829 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:47.264829 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:47.264829 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:47.264829 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:47.264829 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:47.264829 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.264829 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:47.265424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.265424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:47.265424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:47.265424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:47.265424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.265424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:23:47.265424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:47.265424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.265424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:47.265424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:47.265424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:47.265424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.265698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:47.265698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:47.265698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:47.265698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:47.265698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:47.265698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:47.265698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:47.265698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:47.265698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:47.265698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:47.265698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:47.265698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:47.265698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:47.265920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.265920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:47.266078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.266078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:47.266078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:47.266078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:47.266078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:47.266078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:47.266078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.266078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:47.266078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:47.266078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.266078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:47.266078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:47.266078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.266078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:47.266078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:47.266078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:47.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:47.266709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:47.266709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:47.266709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:51.778137 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:51.779863 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:51.780008 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:51.780663 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:51.805206 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:51.805206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:51.805487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:51.805487 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:51.805487 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:51.805487 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:51.805487 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:51.805487 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:51.805487 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:51.805487 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:51.806419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:51.806419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:51.806419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:51.806419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:51.806419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:51.806419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:23:51.806419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:51.806419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:51.806419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:51.806419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:51.806419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:51.806419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:51.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:51.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:51.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:51.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:51.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:51.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:51.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:51.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:51.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:51.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:51.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:51.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:51.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:51.807194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:51.807194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:51.807561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:51.807561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:51.807561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:51.807561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:51.807561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:51.807561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:51.807561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:51.807561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:51.807561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:51.807561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:51.807561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:51.807561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:51.807561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:51.807561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:51.807561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:51.807561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:51.808025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:51.808340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:51.808340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:51.808340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:52.233641 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:52.234505 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:52.234505 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:52.235619 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:52.235876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.236484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:52.236484 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:52.236484 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:52.236484 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:52.236484 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:52.236484 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:52.236484 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.236484 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:52.237510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.237510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:52.237510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:52.237510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:52.237510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.237510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:23:52.237510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:52.237510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.237510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:52.237510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:52.237510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:52.237510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.237804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:52.237804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:52.237804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:52.237804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:52.237804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:52.237804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:52.237804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:52.237804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:52.237804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:52.237804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:52.237804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:52.237804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:52.237804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:52.238049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.238049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:52.238241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.238241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:52.238241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:52.238241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:52.238241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:52.238241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:52.238241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.238241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:52.238241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:52.238241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.238241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:52.238241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:52.238241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.238241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:52.238241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:52.238241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:52.238521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:52.238894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:52.238894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.238894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:52.243134 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:52.243400 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:52.243487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:52.243600 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:23:52.244033 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:52.244275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.244746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:52.244746 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:52.244746 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:52.244746 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:52.244746 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:52.244746 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:52.244746 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.244746 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:52.245233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.245233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:23:52.245233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:52.245233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:52.245233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.245233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:52.245233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:52.245233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:23:52.245233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.245233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:52.245233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:52.245233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:23:52.245233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:52.245233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:23:52.245233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:23:52.245504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:52.245504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:23:52.245504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:52.245504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:23:52.245504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:23:52.245504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:52.245504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:23:52.245504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.245504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:23:52.245777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.245777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:52.245777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:52.245777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:52.245777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.245777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:23:52.245777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:23:52.245777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:23:52.245777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:52.245777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:52.245777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.245777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:52.246023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:52.246023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:52.246023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.246023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:52.246023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:52.246023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:52.246023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:52.246023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:52.246023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:52.246023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:52.246023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:52.246023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:52.246023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:52.246023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:52.246272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:52.246272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:52.246272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.246272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:52.246631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.246631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:52.246631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:52.246631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:52.246631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:52.246631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:52.246631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.246631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:52.246631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:52.246631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.246631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:52.246631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:52.246631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.246631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:52.246631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:52.246631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:52.246907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:52.247335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:52.247335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:52.247335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:52.247335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:52.247335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:52.247335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.247335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:52.252276 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:52.252580 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:52.252673 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:52.252772 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:52.253186 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:52.253486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.254941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:52.254941 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:52.254941 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:52.254941 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:52.254941 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:52.254941 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:52.254941 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.254941 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:52.255467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.255467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:23:52.255467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:52.255467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:52.255467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.255467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:52.255467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:52.255467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:23:52.255467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.255467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:52.255467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:52.255467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:23:52.255467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:52.255467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:23:52.255467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:23:52.255786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:52.255786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:23:52.255786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:52.255786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:23:52.255786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:23:52.255786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:52.255786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:23:52.255786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.255786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:23:52.256071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.256071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:52.256071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:52.256071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:52.256071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.256071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:23:52.256071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:23:52.256071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:52.256071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:52.256071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.256071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:52.256299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:52.256299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:52.256299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.256299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:52.256299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:52.256299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:52.256299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:52.256299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:52.256299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:52.256299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:52.256299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:52.256299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:52.256299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:52.256299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:52.256556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:52.256556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:52.256556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.256556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.256999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:52.257348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:52.257704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:52.257704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:52.257704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.257704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:52.260826 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:52.261062 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:52.261144 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:52.261266 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:52.261680 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:52.261935 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:52.262049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.262493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:52.262493 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:52.262493 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:52.262493 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:52.262493 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:52.262493 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:52.262493 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.262493 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:52.263058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.263058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:52.263058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:52.263058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:52.263058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.263058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:23:52.263058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:52.263058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.263058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:52.263058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:52.263058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:52.263058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.263297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:52.263297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:52.263297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:52.263297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:52.263297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:52.263297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:52.263297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:52.263297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:52.263297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:52.263297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:52.263297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:52.263297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:52.263297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:52.263508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.263508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:52.263705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.263705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:52.263705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:52.263705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:52.263705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:52.263705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:52.263705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.263705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:52.263705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:52.263705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.263705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:52.263705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:52.263705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.263705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:52.263705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:52.263705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:52.264009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:52.264338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:52.264338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:52.264338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:56.779988 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:56.781606 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:56.781606 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:56.781606 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:56.806217 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:56.806379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:56.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:56.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:56.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:56.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:56.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:56.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:56.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:56.806817 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:56.807419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:56.807419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:56.807419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:56.807419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:56.807419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:56.807419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:23:56.807419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:56.807419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:56.807419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:56.807419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:56.807419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:56.807419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:56.807798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:56.807798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:56.807798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:56.807798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:56.807798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:56.807798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:56.807798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:56.807798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:56.807798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:56.807798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:56.807798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:56.807798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:56.807798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:56.807994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:56.807994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:56.808120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:56.808120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:56.808120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:56.808120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:56.808120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:56.808120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:56.808120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:56.808120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:56.808120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:56.808120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:56.808120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:56.808120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:56.808120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:56.808120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:56.808120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:56.808120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:56.808400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:56.808709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:56.808709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:56.808709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:57.234119 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:57.234516 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:57.234789 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:57.235340 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:57.235488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.235903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:57.235903 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:57.235903 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:57.235903 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:57.235903 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:57.235903 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:57.235903 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.235903 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:57.236532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.236532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:57.236532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:57.236532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:57.236532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.236532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:23:57.236532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:57.236532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.236532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:57.236532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:57.236532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:57.236532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.236771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:57.236771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:57.236771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:57.236771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:57.236771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:57.236771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:57.236771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:57.236771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:57.236771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:57.236771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:57.236771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:57.236771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:57.236771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:57.236995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.236995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:57.237658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.237658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:57.237658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:57.237658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:57.237658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:57.237658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:57.237658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.237658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:57.237658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:57.237658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.237658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:57.237658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:57.237658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.237658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:57.237658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:57.237658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:57.237960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:57.238257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:57.238257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.238257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:57.243593 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:57.243890 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:57.244006 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:57.244106 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:23:57.244405 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:57.244572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.244916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:57.244916 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:57.244916 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:57.244916 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:57.244916 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:57.244916 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:57.244916 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.244916 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:57.245443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.245443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:23:57.245443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:57.245443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:57.245443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.245443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:57.245443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:57.245443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:23:57.245443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.245443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:57.245443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:57.245443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:23:57.245443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:57.245443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:23:57.245443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:23:57.245784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:57.245784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:23:57.245784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:57.245784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:23:57.245784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:23:57.245784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:57.245784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:23:57.245784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.245784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:23:57.246324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.246324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:57.246324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:57.246324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:57.246324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.246324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:23:57.246324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:23:57.246324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:23:57.246324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:57.246324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:57.246324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.246324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:57.246574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:57.246574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:57.246574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.246574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:57.246574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:57.246574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:57.246574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:57.246574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:57.246574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:57.246574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:57.246574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:57.246574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:57.246574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:57.246574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:57.246813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:57.246813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:57.246813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.246813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:57.247469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.247469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:57.247469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:57.247469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:57.247469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:57.247469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:57.247469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.247469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:57.247469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:57.247469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.247469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:57.247469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:57.247469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.247469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:57.247469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:57.247469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:57.247805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:57.248086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:57.248086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:57.248086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:57.248086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:57.248086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:57.248086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.248086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:57.252785 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:57.253091 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:57.253210 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:57.253303 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:57.253630 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:57.253805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.254206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:57.254206 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:57.254206 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:57.254206 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:57.254206 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:57.254206 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:57.254206 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.254206 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:57.254843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.254843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:23:57.254843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:57.254843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:57.254843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.254843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:57.254843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:57.254843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:23:57.254843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.254843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:23:57.254843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:57.254843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:23:57.254843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:57.254843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:23:57.254843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:23:57.255428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:57.255428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:23:57.255428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:57.255428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:23:57.255428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:23:57.255428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:57.255428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:23:57.255428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.255428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:23:57.255624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.255624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:57.255624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:57.255624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:57.255624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.255624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:23:57.255624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:23:57.255624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:57.255624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:57.255624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.255624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:57.255808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:57.255808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:57.255808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.255808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:57.255808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:57.255808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:57.255808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:57.255808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:57.255808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:57.255808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:57.255808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:57.255808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:57.255808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:57.255808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:57.256036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:57.256036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:57.256036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.256036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.256260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:57.256715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:57.257035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:57.257035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:57.257035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.257035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:23:57.261258 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:23:57.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:23:57.261692 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:23:57.261853 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:23:57.262202 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:57.262492 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:23:57.262643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.263141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:23:57.263141 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:57.263141 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:23:57.263141 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:23:57.263141 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:57.263141 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:23:57.263141 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.263141 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:23:57.263894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.263894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:23:57.263894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:57.263894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:23:57.263894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.263894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:23:57.263894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:57.263894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.263894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:57.263894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:57.263894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:23:57.263894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.264257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:23:57.264257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:57.264257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:23:57.264257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:57.264257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:23:57.264257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:23:57.264257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:57.264257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:23:57.264257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:57.264257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:23:57.264257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:23:57.264257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:57.264257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:23:57.264665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.264665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:23:57.264925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.264925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:23:57.264925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:23:57.264925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:23:57.264925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:23:57.264925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:23:57.264925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.264925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:23:57.264925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:23:57.264925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.264925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:23:57.264925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:23:57.264925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.264925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:23:57.264925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:23:57.264925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:23:57.265418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:23:57.266020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:23:57.266020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:23:57.266020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:01.780820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:01.781198 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:01.781277 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:01.782320 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:01.810870 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:01.811100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:01.811586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:01.811586 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:01.811586 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:01.811586 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:01.811586 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:01.811586 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:01.811586 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:01.811586 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:01.812377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:01.812377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:01.812377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:01.812377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:01.812377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:01.812377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:01.812377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:01.812377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:01.812377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:01.812377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:01.812377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:01.812377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:01.813200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:01.813200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:01.813200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:01.813200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:01.813200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:01.813200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:01.813200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:01.813200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:01.813200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:01.813200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:01.813200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:01.813200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:01.813200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:01.813595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:01.813595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:01.813595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:01.813595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:01.813595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:01.813595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:01.813595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:01.813595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:01.813595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:01.813595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:01.813595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:01.813595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:01.813595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:01.813595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:01.813595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:01.813884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:01.814186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:01.814186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:01.814186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:01.814186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:01.814186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:01.814186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:02.234610 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:02.234962 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:02.235074 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:02.235478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:02.235633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.236032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:02.236032 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:02.236032 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:02.236032 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:02.236032 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:02.236032 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:02.236032 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.236032 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:02.236647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.236647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:02.236647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:02.236647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:02.236647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.236647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:02.236647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:02.236647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.236647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:02.236647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:02.236647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:02.236647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.236910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:02.236910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:02.236910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:02.236910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:02.236910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:02.236910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:02.236910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:02.236910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:02.236910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:02.236910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:02.236910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:02.236910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:02.236910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:02.237120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.237120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:02.237353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.237353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:02.237353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:02.237353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:02.237353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:02.237353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:02.237353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.237353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:02.237353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:02.237353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.237353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:02.237353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:02.237353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.237353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:02.237353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:02.237353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:02.237711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:02.238030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:02.238030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.238030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:02.244023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:02.244281 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:02.244365 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:02.244455 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:24:02.244711 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:02.244847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.245181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:02.245181 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:02.245181 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:02.245181 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:02.245181 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:02.245181 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:02.245181 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.245181 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:02.245730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.245730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:02.245730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:02.245730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:02.245730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.245730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:02.245730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:02.245730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:02.245730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.245730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:02.245730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:02.245730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:02.245730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:02.245730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:02.245730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:02.246265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:02.246265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:02.246265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:02.246265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:02.246265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:02.246265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:02.246265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:02.246265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.246265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:02.246265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.246265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:02.246265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:02.246265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:02.246265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.246684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:02.246684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:02.246684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:24:02.246684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:02.246684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:02.246684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.246684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:02.246684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:02.246684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:02.246684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.246684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:02.246684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:02.246684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:02.246924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:02.246924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:02.246924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:02.246924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:02.246924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:02.246924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:02.246924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:02.246924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:02.246924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:02.246924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:02.246924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.246924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:02.247121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.247121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:02.247121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:02.247121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:02.247121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:02.247121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:02.247121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.247121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:02.247121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:02.247121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.247121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:02.247121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:02.247121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.247121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:02.247121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:02.247121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:02.247424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:02.247767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:02.247767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:02.247767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:02.247767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:02.247767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:02.247767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.247767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:02.253206 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:02.253449 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:02.253522 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:02.253643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:02.253865 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:02.254054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.254375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:02.254375 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:02.254375 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:02.254375 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:02.254375 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:02.254375 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:02.254375 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.254375 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:02.254839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.254839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:02.254839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:02.254839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:02.254839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.254839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:02.254839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:02.254839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:02.254839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.254839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:02.254839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:02.254839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:02.254839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:02.254839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:02.254839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:02.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:02.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:02.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:02.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:02.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:02.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:02.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:02.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.255262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:02.255660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.255660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:02.255660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:02.255660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:02.255660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.255660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:02.255660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:02.255660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:02.255660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:02.255660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.255660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:02.255986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:02.255986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:02.255986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.255986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:02.255986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:02.255986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:02.255986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:02.255986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:02.255986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:02.255986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:02.255986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:02.255986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:02.255986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:02.255986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:02.256379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:02.256379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:02.256379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.256379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:02.256379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.256379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:02.256379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:02.256379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:02.256379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:02.256379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:02.256379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.256379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:02.256379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:02.256379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:02.256810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:02.257473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:02.257473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:02.257473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:02.257473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:02.257473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:02.257473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:02.257473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:02.257473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:02.257473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:02.257473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.257473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:02.261723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:02.261973 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:02.262082 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:02.262205 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:02.262449 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:02.262662 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:02.262768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.263108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:02.263108 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:02.263108 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:02.263108 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:02.263108 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:02.263108 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:02.263108 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.263108 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:02.263660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.263660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:02.263660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:02.263660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:02.263660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.263660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:02.263660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:02.263660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.263660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:02.263660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:02.263660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:02.263660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.264028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:02.264028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:02.264028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:02.264028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:02.264028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:02.264028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:02.264028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:02.264028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:02.264028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:02.264028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:02.264028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:02.264028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:02.264028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:02.264390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.264390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:02.264390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.264390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:02.264390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:02.264390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:02.264390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:02.264390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:02.264390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.264390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:02.264390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:02.264390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.264390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:02.264390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:02.264390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:02.264818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:02.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:02.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:02.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:02.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:02.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:02.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:06.781201 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:06.781594 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:06.781718 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:06.782364 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:06.807233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:06.807456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:06.807905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:06.807905 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:06.807905 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:06.807905 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:06.807905 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:06.807905 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:06.807905 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:06.807905 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:06.808657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:06.808657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:06.808657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:06.808657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:06.808657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:06.808657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:06.808657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:06.808657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:06.808657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:06.808657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:06.808657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:06.808657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:06.808926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:06.808926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:06.808926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:06.808926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:06.808926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:06.808926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:06.808926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:06.808926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:06.808926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:06.808926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:06.808926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:06.808926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:06.808926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:06.809160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:06.809160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:06.809403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:06.809403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:06.809403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:06.809403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:06.809403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:06.809403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:06.809403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:06.809403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:06.809403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:06.809403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:06.809403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:06.809403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:06.809403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:06.809403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:06.809403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:06.809403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:06.809736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:06.810074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:06.810074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:06.810074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:07.235020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:07.235263 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:07.235351 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:07.235805 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:07.235966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.236603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:07.236603 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:07.236603 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:07.236603 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:07.236603 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:07.236603 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:07.236603 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.236603 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:07.237278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.237278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:07.237278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:07.237278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:07.237278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.237278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:07.237278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:07.237278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.237278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:07.237278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:07.237278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:07.237278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.237623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:07.237623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:07.237623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:07.237623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:07.237623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:07.237623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:07.237623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:07.237623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:07.237623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:07.237623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:07.237623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:07.237623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:07.237623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:07.237915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.237915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:07.238273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.238273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:07.238273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:07.238273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:07.238273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:07.238273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:07.238273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.238273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:07.238273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:07.238273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.238273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:07.238273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:07.238273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.238273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:07.238273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:07.238273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:07.238699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:07.239033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:07.239033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.239033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:07.244433 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:07.244688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:07.244767 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:07.244860 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:24:07.245142 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:07.245317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.245761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:07.245761 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:07.245761 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:07.245761 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:07.245761 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:07.245761 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:07.245761 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.245761 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:07.246954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.246954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:07.246954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:07.246954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:07.246954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.246954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:07.246954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:07.246954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:07.246954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.246954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:07.246954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:07.246954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:07.246954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:07.246954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:07.246954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:07.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:07.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:07.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:07.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:07.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:07.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:07.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:07.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:07.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:07.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:07.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:07.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.248016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:07.248016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:07.248016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:24:07.248016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:07.248016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:07.248016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.248016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:07.248016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:07.248016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:07.248016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.248016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:07.248016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:07.248016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:07.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:07.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:07.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:07.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:07.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:07.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:07.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:07.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:07.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:07.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:07.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:07.248824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.248824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:07.248824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:07.248824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:07.248824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:07.248824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:07.248824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.248824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:07.248824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:07.248824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.248824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:07.248824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:07.248824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.248824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:07.248824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:07.248824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:07.249283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:07.249811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:07.249811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:07.249811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:07.249811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:07.249811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:07.249811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.249811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:07.253497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:07.253739 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:07.253836 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:07.253968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:07.254268 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:07.254466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.254846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:07.254846 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:07.254846 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:07.254846 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:07.254846 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:07.254846 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:07.254846 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.254846 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:07.255406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.255406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:07.255406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:07.255406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:07.255406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.255406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:07.255406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:07.255406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:07.255406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.255406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:07.255406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:07.255406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:07.255406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:07.255406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:07.255406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:07.255815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:07.255815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:07.255815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:07.255815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:07.255815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:07.255815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:07.255815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:07.255815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.255815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:07.256240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.256240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:07.256240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:07.256240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:07.256240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.256240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:07.256240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:07.256240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:07.256240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:07.256240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.256240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:07.256554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:07.256554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:07.256554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.256554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:07.256554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:07.256554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:07.256554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:07.256554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:07.256554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:07.256554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:07.256554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:07.256554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:07.256554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:07.256554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:07.256835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:07.256835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:07.256835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.256835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.257109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:07.257588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:07.258077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:07.258077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:07.258077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.258077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:07.262057 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:07.262303 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:07.262403 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:07.262618 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:07.262826 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:07.263004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:07.263103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.263514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:07.263514 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:07.263514 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:07.263514 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:07.263514 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:07.263514 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:07.263514 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.263514 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:07.264086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.264086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:07.264086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:07.264086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:07.264086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.264086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:07.264086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:07.264086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.264086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:07.264086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:07.264086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:07.264086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.264429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:07.264429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:07.264429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:07.264429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:07.264429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:07.264429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:07.264429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:07.264429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:07.264429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:07.264429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:07.264429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:07.264429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:07.264429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:07.264817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.264817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:07.264881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.264881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:07.264881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:07.264881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:07.264881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:07.264881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:07.264881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.264881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:07.264881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:07.264881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.264881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:07.264881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:07.264881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.264881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:07.264881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:07.264881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:07.265288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:07.265810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:07.265810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:07.265810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:11.781727 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:11.782200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:11.782231 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:11.783015 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:11.809056 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:11.809228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:11.809666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:11.809666 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:11.809666 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:11.809666 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:11.809666 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:11.809666 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:11.809666 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:11.809666 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:11.810343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:11.810343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:11.810343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:11.810343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:11.810343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:11.810343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:11.810343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:11.810343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:11.810343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:11.810343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:11.810343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:11.810343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:11.810587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:11.810587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:11.810587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:11.810587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:11.810587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:11.810587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:11.810587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:11.810587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:11.810587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:11.810587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:11.810587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:11.810587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:11.810587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:11.810823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:11.810823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:11.811093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:11.811093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:11.811093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:11.811093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:11.811093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:11.811093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:11.811093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:11.811093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:11.811093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:11.811093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:11.811093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:11.811093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:11.811093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:11.811093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:11.811093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:11.811093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:11.811446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:11.811756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:11.811756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:11.811756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:12.235634 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:12.236014 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:12.236198 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:12.236593 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:12.236768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.237162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:12.237162 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:12.237162 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:12.237162 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:12.237162 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:12.237162 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:12.237162 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.237162 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:12.237817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.237817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:12.237817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:12.237817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:12.237817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.237817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:12.237817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:12.237817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.237817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:12.237817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:12.237817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:12.237817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.238354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:12.238354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:12.238354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:12.238354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:12.238354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:12.238354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:12.238354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:12.238354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:12.238354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:12.238354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:12.238354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:12.238354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:12.238354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:12.238579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.238579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:12.238579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.238579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:12.238579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:12.238579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:12.238579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:12.238579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:12.238579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.238579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:12.238579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:12.238579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.238579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:12.238579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:12.238579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:12.238832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:12.239136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:12.239136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:12.239136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:12.239136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:12.239136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.239136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:12.244781 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:12.244982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:12.245133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:12.245265 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:24:12.245413 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:12.245586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.245979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:12.245979 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:12.245979 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:12.245979 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:12.245979 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:12.245979 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:12.245979 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.245979 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:12.246500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.246500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:12.246500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:12.246500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:12.246500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.246500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:12.246500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:12.246500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:12.246500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.246500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:12.246500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:12.246500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:12.246500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:12.246500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:12.246500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:12.246782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:12.246782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:12.246782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:12.246782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:12.246782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:12.246782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:12.246782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:12.246782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.246782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:12.247077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.247077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:12.247077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:12.247077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:12.247077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.247077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:12.247077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:12.247077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:24:12.247077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:12.247077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:12.247077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.247077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:12.247324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:12.247324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:12.247324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.247324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:12.247324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:12.247324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:12.247324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:12.247324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:12.247324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:12.247324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:12.247324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:12.247324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:12.247324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:12.247324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:12.247526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:12.247526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:12.247526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.247526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:12.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:12.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:12.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:12.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:12.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:12.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:12.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:12.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:12.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:12.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:12.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:12.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:12.248878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:12.249168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:12.249168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:12.249168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:12.249168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:12.249168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:12.249168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.249168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:12.253869 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:12.254597 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:12.254597 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:12.254597 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:12.254936 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:12.255209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.255658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:12.255658 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:12.255658 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:12.255658 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:12.255658 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:12.255658 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:12.255658 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.255658 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:12.256403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.256403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:12.256403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:12.256403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:12.256403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.256403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:12.256403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:12.256403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:12.256403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.256403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:12.256403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:12.256403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:12.256403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:12.256403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:12.256403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:12.258190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:12.258190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:12.258190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:12.258190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:12.258190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:12.258190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:12.258190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:12.258190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.258190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:12.258190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.258190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:12.258190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:12.258190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:12.258190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.258442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:12.258442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:12.258442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:12.258442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:12.258442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.258442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:12.258442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:12.258442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:12.258442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.258442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:12.258442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:12.258442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:12.258442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:12.258678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:12.258678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:12.258678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:12.258678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:12.258678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:12.258678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:12.258678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:12.258678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:12.258678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:12.258678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.258678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:12.258678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:12.258923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.259246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:12.262411 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:12.262674 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:12.262837 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:12.263034 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:12.263326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:12.263587 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:12.263735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.264190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:12.264190 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:12.264190 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:12.264190 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:12.264190 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:12.264190 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:12.264190 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.264190 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:12.264924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.264924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:12.264924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:12.264924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:12.264924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.264924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:12.264924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:12.264924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.264924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:12.264924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:12.264924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:12.264924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.265189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:12.265189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:12.265189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:12.265189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:12.265189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:12.265189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:12.265189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:12.265189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:12.265189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:12.265189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:12.265189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:12.265189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:12.265189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:12.265415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.265415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:12.266155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.266155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:12.266155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:12.266155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:12.266155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:12.266155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:12.266155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.266155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:12.266155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:12.266155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.266155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:12.266155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:12.266155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.266155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:12.266155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:12.266155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:12.266483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:12.266839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:12.266839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:12.266839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:16.782159 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:16.782670 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:16.782670 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:16.783215 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:16.784267 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=596,neutron:INSERT=1,neutron:UPDATE=2,neutron:DELETE=1 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:24:16.809461 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:16.809643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:16.810072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:16.810072 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:16.810072 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:16.810072 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:16.810072 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:16.810072 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:16.810072 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:16.810072 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:16.810781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:16.810781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:16.810781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:16.810781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:16.810781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:16.810781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:16.810781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:16.810781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:16.810781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:16.810781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:16.810781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:16.810781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:16.811007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:16.811007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:16.811007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:16.811007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:16.811007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:16.811007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:16.811007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:16.811007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:16.811007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:16.811007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:16.811007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:16.811007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:16.811007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:16.811222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:16.811222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:16.811548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:16.811548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:16.811548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:16.811548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:16.811548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:16.811548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:16.811548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:16.811548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:16.811548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:16.811548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:16.811548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:16.811548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:16.811548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:16.811548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:16.811548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:16.811548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:16.811825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:16.812173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:16.812173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:16.812173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:17.236159 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:17.236497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:17.236689 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:17.237044 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:17.237187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.237599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:17.237599 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:17.237599 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:17.237599 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:17.237599 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:17.237599 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:17.237599 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.237599 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:17.238227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.238227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:17.238227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:17.238227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:17.238227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.238227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:17.238227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:17.238227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.238227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:17.238227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:17.238227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:17.238227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.238482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:17.238482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:17.238482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:17.238482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:17.238482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:17.238482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:17.238482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:17.238482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:17.238482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:17.238482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:17.238482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:17.238482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:17.238482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:17.238690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.238690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:17.238938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.238938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:17.238938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:17.238938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:17.238938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:17.238938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:17.238938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.238938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:17.238938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:17.238938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.238938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:17.238938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:17.238938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.238938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:17.238938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:17.238938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:17.239214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:17.239567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:17.239567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.239567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:17.245094 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:17.245335 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:17.245417 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:17.245512 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:24:17.245758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:17.245899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.246213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:17.246213 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:17.246213 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:17.246213 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:17.246213 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:17.246213 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:17.246213 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.246213 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:17.246764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.246764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:17.246764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:17.246764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:17.246764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.246764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:17.246764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:17.246764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:17.246764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.246764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:17.246764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:17.246764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:17.246764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:17.246764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:17.246764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:17.247009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:17.247009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:17.247009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:17.247009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:17.247009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:17.247009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:17.247009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:17.247009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.247009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:17.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:17.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:17.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:17.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:17.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:17.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:24:17.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:17.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:17.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.247354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:17.247923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:17.247923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:17.247923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.247923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:17.247923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:17.247923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:17.247923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:17.247923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:17.247923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:17.247923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:17.247923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:17.247923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:17.247923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:17.247923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:17.248454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:17.248454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:17.248454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.248454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:17.248454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.248454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:17.248454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:17.248454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:17.248454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:17.248454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:17.248454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.248454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:17.248454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:17.248454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:17.248770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:17.249053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:17.249053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:17.249053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:17.249053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:17.249053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:17.249053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:17.249053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:17.249053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:17.249053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:17.249053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:17.249053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.249053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:17.254223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:17.254496 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:17.254628 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:17.254753 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:17.255085 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:17.255313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.255797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:17.255797 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:17.255797 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:17.255797 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:17.255797 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:17.255797 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:17.255797 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.255797 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:17.256434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.256434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:17.256434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:17.256434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:17.256434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.256434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:17.256434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:17.256434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:17.256434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.256434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:17.256434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:17.256434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:17.256434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:17.256434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:17.256434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:17.256704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:17.256704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:17.256704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:17.256704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:17.256704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:17.256704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:17.256704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:17.256704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.256704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:17.257177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.257177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:17.257177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:17.257177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:17.257177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.257177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:17.257177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:17.257177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:17.257177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:17.257177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.257177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:17.257386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:17.257386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:17.257386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.257386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:17.257386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:17.257386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:17.257386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:17.257386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:17.257386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:17.257386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:17.257386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:17.257386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:17.257386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:17.257386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:17.257617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:17.257617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:17.257617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.257617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.258085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:17.258512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:17.258811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:17.258811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:17.258811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.258811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:17.262688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:17.262853 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:17.262954 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:17.263086 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:17.263341 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:17.263532 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:17.263648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.264045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:17.264045 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:17.264045 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:17.264045 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:17.264045 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:17.264045 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:17.264045 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.264045 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:17.264698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.264698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:17.264698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:17.264698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:17.264698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.264698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:17.264698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:17.264698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.264698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:17.264698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:17.264698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:17.264698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.264944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:17.264944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:17.264944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:17.264944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:17.264944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:17.264944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:17.264944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:17.264944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:17.264944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:17.264944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:17.264944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:17.264944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:17.264944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:17.265134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.265134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:17.265481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.265481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:17.265481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:17.265481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:17.265481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:17.265481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:17.265481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.265481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:17.265481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:17.265481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.265481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:17.265481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:17.265481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.265481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:17.265481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:17.265481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:17.265772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:17.266068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:17.266068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:17.266068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:21.782691 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:21.783157 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:21.783157 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:21.783850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:21.811833 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:21.812074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:21.812684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:21.812684 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:21.812684 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:21.812684 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:21.812684 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:21.812684 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:21.812684 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:21.812684 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:21.813698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:21.813698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:21.813698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:21.813698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:21.813698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:21.813698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:21.813698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:21.813698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:21.813698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:21.813698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:21.813698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:21.813698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:21.813961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:21.813961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:21.813961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:21.813961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:21.813961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:21.813961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:21.813961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:21.813961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:21.813961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:21.813961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:21.813961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:21.813961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:21.813961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:21.814151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:21.814151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:21.814874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:21.814874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:21.814874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:21.814874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:21.814874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:21.814874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:21.814874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:21.814874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:21.814874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:21.814874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:21.814874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:21.814874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:21.814874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:21.814874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:21.814874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:21.814874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:21.815161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:21.815463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:21.815463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:21.815463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:22.236631 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:22.236948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:22.236948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:22.237295 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:22.237397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.237806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:22.237806 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:22.237806 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:22.237806 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:22.237806 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:22.237806 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:22.237806 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.237806 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:22.238379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.238379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:22.238379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:22.238379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:22.238379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.238379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:22.238379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:22.238379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.238379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:22.238379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:22.238379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:22.238379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.238664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:22.238664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:22.238664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:22.238664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:22.238664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:22.238664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:22.238664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:22.238664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:22.238664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:22.238664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:22.238664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:22.238664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:22.238664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:22.238868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.238868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:22.239125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.239125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:22.239125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:22.239125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:22.239125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:22.239125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:22.239125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.239125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:22.239125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:22.239125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.239125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:22.239125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:22.239125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.239125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:22.239125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:22.239125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:22.239958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:22.240279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:22.240279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.240279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:22.245349 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:22.245530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:22.245701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:22.245701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:24:22.245885 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:22.246017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.246310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:22.246310 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:22.246310 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:22.246310 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:22.246310 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:22.246310 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:22.246310 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.246310 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:22.246760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.246760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:22.246760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:22.246760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:22.246760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.246760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:22.246760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:22.246760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:22.246760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.246760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:22.246760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:22.246760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:22.246760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:22.246760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:22.246760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:22.247047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:22.247047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:22.247047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:22.247047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:22.247047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:22.247047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:22.247047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:22.247047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.247047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:22.247503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.247503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:22.247503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:22.247503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:22.247503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.247503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:22.247503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:22.247503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:24:22.247503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:22.247503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:22.247503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.247503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:22.248109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:22.248109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:22.248109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.248109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:22.248109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:22.248109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:22.248109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:22.248109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:22.248109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:22.248109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:22.248109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:22.248109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:22.248109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:22.248109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:22.248379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:22.248379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:22.248379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.248379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:22.248379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.248379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:22.248379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:22.248379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:22.248379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:22.248379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:22.248379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.248379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:22.248379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:22.248379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:22.248633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:22.248949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:22.248949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:22.248949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:22.248949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:22.248949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:22.248949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:22.248949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:22.248949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:22.248949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:22.248949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:22.248949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.248949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:22.254488 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:22.254668 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:22.254748 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:22.254841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:22.255011 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:22.255154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.255452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:22.255452 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:22.255452 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:22.255452 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:22.255452 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:22.255452 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:22.255452 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.255452 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:22.255895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.255895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:22.255895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:22.255895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:22.255895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.255895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:22.255895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:22.255895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:22.255895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.255895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:22.255895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:22.255895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:22.255895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:22.255895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:22.255895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:22.256190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:22.256190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:22.256190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:22.256190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:22.256190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:22.256190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:22.256190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:22.256190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.256190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:22.256613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.256613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:22.256613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:22.256613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:22.256613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.256613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:22.256613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:22.256613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:22.256613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:22.256613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.256613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:22.257125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:22.257125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:22.257125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.257125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:22.257125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:22.257125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:22.257125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:22.257125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:22.257125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:22.257125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:22.257125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:22.257125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:22.257125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:22.257125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:22.257411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:22.257411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:22.257411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.257411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:22.257411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.257411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:22.257411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:22.257411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:22.257411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:22.257411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:22.257411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.257411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:22.257411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:22.257411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:22.257645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:22.257936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:22.257936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:22.257936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:22.257936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:22.257936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:22.257936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:22.257936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:22.257936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:22.257936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:22.257936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.257936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:22.262992 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:22.263193 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:22.263297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:22.263411 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:22.263645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:22.263799 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:22.263875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.264206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:22.264206 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:22.264206 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:22.264206 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:22.264206 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:22.264206 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:22.264206 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.264206 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:22.264781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.264781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:22.264781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:22.264781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:22.264781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.264781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:22.264781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:22.264781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.264781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:22.264781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:22.264781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:22.264781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.265016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:22.265016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:22.265016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:22.265016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:22.265016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:22.265016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:22.265016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:22.265016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:22.265016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:22.265016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:22.265016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:22.265016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:22.265016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:22.265207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.265207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:22.265726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.265726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:22.265726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:22.265726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:22.265726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:22.265726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:22.265726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.265726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:22.265726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:22.265726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.265726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:22.265726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:22.265726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.265726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:22.265726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:22.265726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:22.266024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:22.266356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:22.266356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:22.266356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:26.783489 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:26.783998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:26.783998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:26.784746 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:26.812445 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:26.812726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:26.815073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:26.815073 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:26.815073 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:26.815073 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:26.815073 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:26.815073 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:26.815073 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:26.815073 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:26.815073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:26.815073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:26.815073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:26.815073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:26.815073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:26.815073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:26.815073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:26.815435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:26.815435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:26.815435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:26.815435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:26.815435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:26.815435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:26.815435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:26.815435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:26.815435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:26.815435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:26.815435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:26.815435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:26.815435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:26.815435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:26.815699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:26.815699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:26.815699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:26.815699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:26.815699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:26.815699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:26.815699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:26.815699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:26.815699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:26.815699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:26.815699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:26.815699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:26.815699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:26.815699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:26.815961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:26.816373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:26.816373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:26.816373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:26.816373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:26.816373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:26.816373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:26.816373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:26.816373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:26.816373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:27.237015 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:27.237955 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:27.237955 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:27.238262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:27.238471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.240848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:27.240848 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:27.240848 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:27.240848 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:27.240848 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:27.240848 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:27.240848 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.240848 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:27.240848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.240848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:27.240848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:27.240848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:27.240848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.240848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:27.240848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:27.241265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.241265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:27.241265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:27.241265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:27.241265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.241265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:27.241265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:27.241265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:27.241265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:27.241265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:27.241265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:27.241265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:27.241265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:27.241265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:27.241608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:27.241608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:27.241608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:27.241608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:27.241608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.241608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:27.241608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.241608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:27.241608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:27.241608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:27.241608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:27.241608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:27.241608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.241608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:27.241961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:27.242449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:27.242449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:27.242449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:27.242449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:27.242449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:27.242449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:27.242449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:27.242449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:27.242449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.242449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:27.245671 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:27.245943 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:27.246128 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:27.246318 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:24:27.246729 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:27.246892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.249573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:27.250035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:27.250035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:27.250035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:27.250035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:27.250035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:27.250035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:27.250035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:27.250035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:27.250035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:27.250035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:27.250035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:27.250035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:27.250035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.250035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:27.250035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:27.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:27.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:27.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:27.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:27.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:24:27.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:27.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:27.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:27.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:27.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:27.250647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.250647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:27.250647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:27.250647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:27.250647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:27.250647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:27.250647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:27.250647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:27.250647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:27.250647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:27.250647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:27.250647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:27.250647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:27.250647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:27.250864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.250864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:27.250864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.250864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:27.250864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:27.250864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:27.250864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:27.250864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:27.250864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.250864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:27.250864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:27.250864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.250864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:27.250864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:27.250864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:27.251119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:27.251453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:27.251453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:27.251453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:27.251453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:27.251453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:27.251453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:27.251453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:27.251453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.251453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:27.254809 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:27.255060 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:27.255165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:27.255298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:27.255612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:27.255803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.256200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:27.256200 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:27.256200 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:27.256200 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:27.256200 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:27.256200 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:27.256200 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.256200 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:27.256853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.256853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:27.256853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:27.256853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:27.256853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.256853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:27.256853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:27.256853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:27.256853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.256853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:27.256853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:27.256853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:27.256853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:27.256853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:27.256853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:27.257344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:27.257344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:27.257344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:27.257344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:27.257344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:27.257344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:27.257344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:27.257344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.257344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:27.257785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.257785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:27.257785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:27.257785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:27.257785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.257785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:27.257785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:27.257785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:27.257785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:27.257785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.257785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:27.258127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:27.258127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:27.258127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.258127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:27.258127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:27.258127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:27.258127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:27.258127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:27.258127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:27.258127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:27.258127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:27.258127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:27.258127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:27.258127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:27.258533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:27.258533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:27.258533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.258533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:27.258533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.258533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:27.258533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:27.258533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:27.258533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:27.258533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:27.258533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.258533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:27.258533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:27.258533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:27.258959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:27.259503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:27.259503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:27.259503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:27.259503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:27.259503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:27.259503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:27.259503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:27.259503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:27.259503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:27.259503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.259503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:27.263374 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:27.263642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:27.263742 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:27.263891 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:27.264147 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:27.264393 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:27.264516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.264956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:27.264956 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:27.264956 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:27.264956 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:27.264956 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:27.264956 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:27.264956 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.264956 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:27.265670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.265670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:27.265670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:27.265670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:27.265670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.265670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:27.265670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:27.265670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.265670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:27.265670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:27.265670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:27.265670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.265930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:27.265930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:27.265930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:27.265930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:27.265930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:27.265930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:27.265930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:27.265930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:27.265930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:27.265930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:27.265930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:27.265930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:27.265930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:27.266207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.266207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:27.266476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.266476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:27.266476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:27.266476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:27.266476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:27.266476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:27.266476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.266476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:27.266476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:27.266476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.266476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:27.266476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:27.266476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.266476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:27.266476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:27.266476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:27.266899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:27.267456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:27.267456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:27.267456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:31.783964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:31.784487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:31.784487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:31.785320 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:31.817992 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:31.818207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:31.818861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:31.818861 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:31.818861 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:31.818861 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:31.818861 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:31.818861 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:31.818861 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:31.818861 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:31.819935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:31.819935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:31.819935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:31.819935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:31.819935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:31.819935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:31.819935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:31.819935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:31.819935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:31.819935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:31.819935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:31.819935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:31.820277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:31.820277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:31.820277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:31.820277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:31.820277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:31.820277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:31.820277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:31.820277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:31.820277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:31.820277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:31.820277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:31.820277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:31.820277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:31.820616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:31.820616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:31.821032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:31.821032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:31.821032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:31.821032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:31.821032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:31.821032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:31.821032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:31.821032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:31.821032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:31.821032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:31.821032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:31.821032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:31.821032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:31.821032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:31.821032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:31.821032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:31.821461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:31.821909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:31.821909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:31.821909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:32.237707 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:32.238043 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:32.238162 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:32.238826 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:32.239066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.239552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:32.239552 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:32.239552 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:32.239552 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:32.239552 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:32.239552 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:32.239552 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.239552 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:32.240474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.240474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:32.240474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:32.240474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:32.240474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.240474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:32.240474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:32.240474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.240474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:32.240474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:32.240474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:32.240474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.240772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:32.240772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:32.240772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:32.240772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:32.240772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:32.240772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:32.240772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:32.240772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:32.240772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:32.240772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:32.240772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:32.240772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:32.240772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:32.240961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.240961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:32.241485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.241485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:32.241485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:32.241485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:32.241485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:32.241485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:32.241485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.241485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:32.241485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:32.241485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.241485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:32.241485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:32.241485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.241485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:32.241485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:32.241485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:32.241851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:32.242140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:32.242140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.242140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:32.246096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:32.246356 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:32.246440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:32.246611 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:24:32.247027 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:32.247225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.247677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:32.247677 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:32.247677 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:32.247677 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:32.247677 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:32.247677 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:32.247677 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.247677 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:32.248190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.248190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:32.248190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:32.248190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:32.248190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.248190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:32.248190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:32.248190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:32.248190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.248190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:32.248190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:32.248190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:32.248190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:32.248190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:32.248190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:32.248485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:32.248485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:32.248485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:32.248485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:32.248485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:32.248485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:32.248485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:32.248485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.248485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:32.248823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.248823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:32.248823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:32.248823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:32.248823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.248823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:32.248823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:32.248823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:24:32.248823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:32.248823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:32.248823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.248823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:32.249341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:32.249341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:32.249341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.249341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:32.249341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:32.249341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:32.249341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:32.249341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:32.249341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:32.249341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:32.249341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:32.249341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:32.249341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:32.249341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:32.249792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:32.249792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:32.249792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.249792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:32.249792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.249792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:32.249792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:32.249792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:32.249792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:32.249792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:32.249792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.249792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:32.249792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:32.249792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:32.250049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:32.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:32.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:32.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:32.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:32.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:32.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:32.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:32.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:32.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:32.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:32.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.250395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:32.255265 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:32.255520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:32.255633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:32.255761 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:32.256104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:32.256362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.256905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:32.256905 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:32.256905 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:32.256905 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:32.256905 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:32.256905 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:32.256905 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.256905 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:32.257455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.257455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:32.257455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:32.257455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:32.257455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.257455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:32.257455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:32.257455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:32.257455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.257455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:32.257455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:32.257455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:32.257455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:32.257455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:32.257455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:32.257843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:32.257843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:32.257843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:32.257843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:32.257843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:32.257843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:32.257843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:32.257843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.257843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:32.258025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.258025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:32.258025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:32.258025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:32.258025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.258025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:32.258025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:32.258025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:32.258025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:32.258025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.258025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:32.258196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:32.258196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:32.258196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.258196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:32.258196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:32.258196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:32.258196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:32.258196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:32.258196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:32.258196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:32.258196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:32.258196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:32.258196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:32.258196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:32.258424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:32.258424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:32.258424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.258424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.258917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:32.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:32.260440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:32.260440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:32.260440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.260440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:32.263766 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:32.263995 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:32.264070 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:32.264196 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:32.264583 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:32.264876 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:32.265034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.265656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:32.265656 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:32.265656 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:32.265656 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:32.265656 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:32.265656 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:32.265656 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.265656 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:32.266507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.266507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:32.266507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:32.266507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:32.266507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.266507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:32.266507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:32.266507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.266507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:32.266507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:32.266507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:32.266507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.267055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:32.267055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:32.267055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:32.267055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:32.267055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:32.267055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:32.267055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:32.267055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:32.267055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:32.267055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:32.267055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:32.267055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:32.267055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:32.267283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.267283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:32.267521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.267521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:32.267521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:32.267521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:32.267521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:32.267521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:32.267521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.267521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:32.267521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:32.267521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.267521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:32.267521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:32.267521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.267521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:32.267521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:32.267521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:32.267813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:32.268110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:32.268110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:32.268110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:36.784404 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:36.784909 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:36.784909 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:36.785375 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:36.808877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:36.809108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:36.809639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:36.809639 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:36.809639 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:36.809639 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:36.809639 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:36.809639 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:36.809639 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:36.809639 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:36.810604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:36.810604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:36.810604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:36.810604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:36.810604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:36.810604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:36.810604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:36.810604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:36.810604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:36.810604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:36.810604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:36.810604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:36.810833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:36.810833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:36.810833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:36.810833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:36.810833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:36.810833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:36.810833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:36.810833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:36.810833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:36.810833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:36.810833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:36.810833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:36.810833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:36.811033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:36.811033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:36.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:36.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:36.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:36.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:36.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:36.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:36.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:36.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:36.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:36.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:36.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:36.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:36.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:36.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:36.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:36.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:36.811636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:36.811923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:36.811923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:36.811923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:37.238099 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:37.238408 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:37.238583 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:37.239239 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:37.239461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.240031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:37.240031 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:37.240031 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:37.240031 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:37.240031 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:37.240031 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:37.240031 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.240031 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:37.240735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.240735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:37.240735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:37.240735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:37.240735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.240735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:37.240735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:37.240735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.240735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:37.240735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:37.240735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:37.240735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.241068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:37.241068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:37.241068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:37.241068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:37.241068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:37.241068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:37.241068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:37.241068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:37.241068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:37.241068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:37.241068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:37.241068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:37.241068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:37.241265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.241265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:37.241563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.241563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:37.241563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:37.241563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:37.241563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:37.241563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:37.241563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.241563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:37.241563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:37.241563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.241563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:37.241563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:37.241563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.241563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:37.241563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:37.241563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:37.242323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:37.242890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:37.242890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.242890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:37.246374 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:37.246570 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:37.246664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:37.246732 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:24:37.247014 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:37.247174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.247615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:37.247615 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:37.247615 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:37.247615 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:37.247615 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:37.247615 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:37.247615 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.247615 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:37.248113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.248113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:37.248113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:37.248113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:37.248113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.248113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:37.248113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:37.248113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:37.248113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.248113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:37.248113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:37.248113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:37.248113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:37.248113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:37.248113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:37.248381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:37.248381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:37.248381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:37.248381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:37.248381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:37.248381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:37.248381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:37.248381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.248381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:37.248714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.248714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:37.248714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:37.248714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:37.248714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.248714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:37.248714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:37.248714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:24:37.248714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:37.248714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:37.248714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.248714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:37.249314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:37.249314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:37.249314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.249314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:37.249314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:37.249314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:37.249314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:37.249314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:37.249314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:37.249314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:37.249314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:37.249314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:37.249314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:37.249314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:37.249684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:37.249684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:37.249684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.249684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:37.249684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.249684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:37.249684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:37.249684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:37.249684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:37.249684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:37.249684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.249684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:37.249684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:37.249684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:37.249915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:37.250216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:37.250216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:37.250216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:37.250216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:37.250216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:37.250216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:37.250216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:37.250216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:37.250216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:37.250216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:37.250216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.250216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:37.255590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:37.255775 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:37.255858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:37.255959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:37.256234 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:37.256422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.256773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:37.256773 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:37.256773 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:37.256773 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:37.256773 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:37.256773 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:37.256773 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.256773 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:37.257291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.257291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:37.257291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:37.257291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:37.257291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.257291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:37.257291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:37.257291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:37.257291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.257291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:37.257291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:37.257291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:37.257291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:37.257291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:37.257291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:37.257687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:37.257687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:37.257687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:37.257687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:37.257687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:37.257687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:37.257687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:37.257687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.257687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:37.257953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.257953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:37.257953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:37.257953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:37.257953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.257953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:37.257953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:37.257953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:37.257953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:37.257953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.257953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:37.258146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:37.258146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:37.258146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.258146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:37.258146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:37.258146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:37.258146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:37.258146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:37.258146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:37.258146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:37.258146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:37.258146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:37.258146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:37.258146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:37.258357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:37.258357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:37.258357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.258357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.258518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:37.258873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:37.259600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:37.259600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:37.259600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.259600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:37.264027 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:37.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:37.264263 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:37.264415 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:37.264745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:37.264928 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:37.265028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.265413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:37.265413 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:37.265413 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:37.265413 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:37.265413 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:37.265413 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:37.265413 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.265413 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:37.265997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.265997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:37.265997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:37.265997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:37.265997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.265997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:37.265997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:37.265997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.265997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:37.265997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:37.265997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:37.265997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.266433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:37.266433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:37.266433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:37.266433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:37.266433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:37.266433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:37.266433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:37.266433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:37.266433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:37.266433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:37.266433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:37.266433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:37.266433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:37.266750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.266750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:37.266750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.266750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:37.266750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:37.266750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:37.266750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:37.266750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:37.266750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.266750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:37.266750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:37.266750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.266750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:37.266750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:37.266750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:37.267011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:37.267296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:37.267296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:37.267296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:37.267296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:37.267296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:37.267296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:41.784824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:41.785261 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:41.785261 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:41.785832 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:41.816474 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:41.816782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:41.817502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:41.817502 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:41.817502 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:41.817502 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:41.817502 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:41.817502 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:41.817502 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:41.817502 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:41.818564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:41.818564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:41.818564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:41.818564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:41.818564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:41.818564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:41.818564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:41.818564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:41.818564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:41.818564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:41.818564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:41.818564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:41.818854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:41.818854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:41.818854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:41.818854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:41.818854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:41.818854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:41.818854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:41.818854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:41.818854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:41.818854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:41.818854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:41.818854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:41.818854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:41.819048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:41.819048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:41.819671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:41.819671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:41.819671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:41.819671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:41.819671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:41.819671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:41.819671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:41.819671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:41.819671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:41.819671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:41.819671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:41.819671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:41.819671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:41.819671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:41.819671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:41.819671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:41.819941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:41.820297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:41.820297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:41.820297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:42.238600 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:42.238879 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:42.238879 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:42.239510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:42.239797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.240404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:42.240404 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:42.240404 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:42.240404 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:42.240404 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:42.240404 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:42.240404 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.240404 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:42.241105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.241105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:42.241105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:42.241105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:42.241105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.241105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:42.241105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:42.241105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.241105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:42.241105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:42.241105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:42.241105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.241405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:42.241405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:42.241405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:42.241405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:42.241405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:42.241405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:42.241405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:42.241405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:42.241405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:42.241405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:42.241405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:42.241405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:42.241405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:42.241666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.241666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:42.241861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.241861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:42.241861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:42.241861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:42.241861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:42.241861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:42.241861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.241861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:42.241861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:42.241861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.241861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:42.241861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:42.241861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.241861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:42.241861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:42.241861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:42.242159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:42.242494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:42.242494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.242494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:42.246643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:42.246826 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:42.246915 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:42.247010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:24:42.247348 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:42.247493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.247897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:42.247897 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:42.247897 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:42.247897 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:42.247897 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:42.247897 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:42.247897 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.247897 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:42.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:42.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:42.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:42.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:42.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:42.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:42.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:42.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:42.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:42.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:42.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:42.248410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:42.248940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:42.248940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:42.248940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:42.248940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:42.248940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:42.248940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:42.248940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:42.248940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.248940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:42.248940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.248940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:42.248940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:42.248940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:42.248940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.249459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:42.249459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:42.249459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:24:42.249459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:42.249459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:42.249459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.249459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:42.249459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:42.249459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:42.249459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.249459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:42.249459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:42.249459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:42.249900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:42.249900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:42.249900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:42.249900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:42.249900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:42.249900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:42.249900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:42.249900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:42.249900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:42.249900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:42.249900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.249900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:42.250118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.250118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:42.250118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:42.250118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:42.250118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:42.250118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:42.250118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.250118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:42.250118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:42.250118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.250118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:42.250118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:42.250118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.250118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:42.250118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:42.250118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:42.250393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:42.250703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:42.250703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:42.250703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:42.250703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:42.250703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:42.250703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.250703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:42.255928 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:42.256154 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:42.256244 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:42.256395 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:42.256739 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:42.256955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.257372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:42.257372 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:42.257372 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:42.257372 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:42.257372 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:42.257372 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:42.257372 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.257372 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:42.257905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.257905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:42.257905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:42.257905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:42.257905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.257905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:42.257905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:42.257905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:42.257905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.257905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:42.257905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:42.257905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:42.257905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:42.257905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:42.257905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:42.258226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:42.258226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:42.258226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:42.258226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:42.258226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:42.258226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:42.258226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:42.258226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.258226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:42.258750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.258750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:42.258750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:42.258750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:42.258750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.258750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:42.258750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:42.258750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:42.258750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:42.258750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.258750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:42.259004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:42.259004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:42.259004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.259004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:42.259004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:42.259004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:42.259004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:42.259004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:42.259004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:42.259004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:42.259004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:42.259004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:42.259004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:42.259004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:42.259244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:42.259244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:42.259244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.259244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.259738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:42.260266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:42.260836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:42.260836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:42.260836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.260836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:42.264435 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:42.264711 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:42.264812 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:42.264952 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:42.265207 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:42.265399 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:42.265494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.265894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:42.265894 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:42.265894 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:42.265894 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:42.265894 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:42.265894 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:42.265894 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.265894 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:42.266558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.266558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:42.266558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:42.266558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:42.266558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.266558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:42.266558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:42.266558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.266558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:42.266558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:42.266558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:42.266558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.266904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:42.266904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:42.266904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:42.266904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:42.266904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:42.266904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:42.266904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:42.266904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:42.266904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:42.266904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:42.266904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:42.266904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:42.266904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:42.267288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.267288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:42.267585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.267585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:42.267585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:42.267585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:42.267585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:42.267585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:42.267585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.267585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:42.267585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:42.267585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.267585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:42.267585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:42.267585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.267585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:42.267585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:42.267585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:42.267889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:42.268177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:42.268177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:42.268177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:46.785292 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:46.786002 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:46.786002 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:46.786381 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:46.812022 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:46.812245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:46.812826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:46.812826 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:46.812826 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:46.812826 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:46.812826 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:46.812826 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:46.812826 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:46.812826 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:46.813517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:46.813517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:46.813517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:46.813517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:46.813517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:46.813517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:46.813517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:46.813517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:46.813517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:46.813517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:46.813517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:46.813517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:46.813792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:46.813792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:46.813792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:46.813792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:46.813792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:46.813792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:46.813792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:46.813792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:46.813792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:46.813792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:46.813792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:46.813792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:46.813792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:46.814025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:46.814025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:46.814297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:46.814297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:46.814297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:46.814297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:46.814297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:46.814297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:46.814297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:46.814297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:46.814297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:46.814297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:46.814297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:46.814297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:46.814297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:46.814297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:46.814297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:46.814297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:46.815180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:46.815639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:46.815639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:46.815639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:47.239111 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:47.240300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:47.240300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:47.240712 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:47.240913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.243736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:47.243736 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:47.243736 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:47.243736 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:47.243736 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:47.243736 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:47.243736 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.243736 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:47.243736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.243736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:47.243736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:47.243736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:47.243736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.243736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:47.243736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:47.244184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.244184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:47.244184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:47.244184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:47.244184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.244184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:47.244184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:47.244184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:47.244184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:47.244184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:47.244184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:47.244184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:47.244184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:47.244184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:47.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:47.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:47.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:47.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:47.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:47.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:47.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:47.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:47.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:47.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:47.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.244577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:47.244975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:47.245503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:47.245503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:47.245503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:47.245503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:47.245503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:47.245503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:47.245503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:47.245503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:47.245503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.245503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:47.246938 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:47.247134 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:47.247275 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:47.247450 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:24:47.248404 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:47.248404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.248641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:47.248641 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:47.248641 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:47.248641 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:47.248641 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:47.248641 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:47.248641 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.248641 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:47.249458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.249458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:47.249458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:47.249458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:47.249458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.249458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:47.249458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:47.249458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:47.249458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.249458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:47.249458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:47.249458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:47.249458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:47.249458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:47.249458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:47.250368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:47.250368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:47.250368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:47.250368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:47.250368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:47.250368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:47.250368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:47.250368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.250368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:47.250368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.250368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:47.250368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:47.250368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:47.250368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.250820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:47.250820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:47.250820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:24:47.250820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:47.250820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:47.250820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.250820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:47.250820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:47.250820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:47.250820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.250820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:47.250820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:47.250820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:47.251068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:47.251068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:47.251068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:47.251068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:47.251068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:47.251068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:47.251068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:47.251068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:47.251068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:47.251068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:47.251068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.251068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:47.251443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.251443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:47.251443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:47.251443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:47.251443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:47.251443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:47.251443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.251443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:47.251443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:47.251443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.251443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:47.251443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:47.251443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.251443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:47.251443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:47.251443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:47.251740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:47.252084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:47.252084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:47.252084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:47.252084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:47.252084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:47.252084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.252084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:47.256232 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:47.256495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:47.256652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:47.256794 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:47.257768 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:47.257768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.257940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:47.257940 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:47.257940 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:47.257940 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:47.257940 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:47.257940 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:47.257940 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.257940 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:47.258849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.258849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:47.258849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:47.258849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:47.258849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.258849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:47.258849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:47.258849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:47.258849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.258849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:47.258849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:47.258849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:47.258849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:47.258849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:47.258849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:47.259109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:47.259109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:47.259109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:47.259109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:47.259109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:47.259109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:47.259109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:47.259109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.259109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:47.259833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.259833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:47.259833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:47.259833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:47.259833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.259833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:47.259833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:47.259833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:47.259833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:47.259833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.259833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:47.260050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:47.260050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:47.260050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.260050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:47.260050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:47.260050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:47.260050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:47.260050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:47.260050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:47.260050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:47.260050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:47.260050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:47.260050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:47.260050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:47.260289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:47.260289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:47.260289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.260289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.261235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:47.261584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:47.261877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:47.261877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:47.261877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.261877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:47.264750 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:47.264957 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:47.265047 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:47.265185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:47.265804 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:47.265804 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:47.265804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.266229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:47.266229 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:47.266229 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:47.266229 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:47.266229 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:47.266229 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:47.266229 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.266229 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:47.266999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.266999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:47.266999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:47.266999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:47.266999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.266999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:47.266999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:47.266999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.266999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:47.266999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:47.266999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:47.266999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.267383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:47.267383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:47.267383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:47.267383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:47.267383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:47.267383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:47.267383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:47.267383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:47.267383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:47.267383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:47.267383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:47.267383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:47.267383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:47.267726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.267726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:47.267807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.267807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:47.267807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:47.267807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:47.267807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:47.267807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:47.267807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.267807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:47.267807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:47.267807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.267807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:47.267807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:47.267807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.267807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:47.267807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:47.267807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:47.268269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:47.268847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:47.268847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:47.268847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:51.785747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:51.786407 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:51.786407 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:51.787088 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:51.815221 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:51.815769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:51.816353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:51.816353 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:51.816353 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:51.816353 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:51.816353 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:51.816353 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:51.816353 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:51.816353 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:51.817175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:51.817175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:51.817175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:51.817175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:51.817175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:51.817175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:51.817175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:51.817175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:51.817175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:51.817175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:51.817175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:51.817175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:51.817775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:51.817775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:51.817775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:51.817775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:51.817775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:51.817775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:51.817775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:51.817775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:51.817775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:51.817775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:51.817775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:51.817775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:51.817775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:51.818064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:51.818064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:51.818064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:51.818064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:51.818064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:51.818064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:51.818064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:51.818064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:51.818064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:51.818064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:51.818064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:51.818064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:51.818064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:51.818064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:51.818064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:51.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:51.818882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:51.818882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:51.818882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:51.818882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:51.818882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:51.818882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:52.240122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:52.240440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:52.240535 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:52.241132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:52.241336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.241810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:52.241810 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:52.241810 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:52.241810 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:52.241810 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:52.241810 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:52.241810 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.241810 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:52.242613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.242613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:52.242613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:52.242613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:52.242613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.242613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:52.242613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:52.242613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.242613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:52.242613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:52.242613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:52.242613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.242957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:52.242957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:52.242957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:52.242957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:52.242957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:52.242957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:52.242957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:52.242957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:52.242957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:52.242957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:52.242957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:52.242957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:52.242957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:52.243163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.243163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:52.243207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.243207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:52.243207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:52.243207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:52.243207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:52.243207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:52.243207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.243207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:52.243207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:52.243207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.243207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:52.243207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:52.243207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.243207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:52.243207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:52.243207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:52.243507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:52.243860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:52.243860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.243860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:52.247273 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:52.247534 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:52.247649 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:52.247734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:24:52.248042 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:52.248208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.248621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:52.248621 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:52.248621 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:52.248621 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:52.248621 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:52.248621 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:52.248621 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.248621 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:52.249211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.249211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:52.249211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:52.249211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:52.249211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.249211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:52.249211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:52.249211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:52.249211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.249211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:52.249211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:52.249211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:52.249211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:52.249211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:52.249211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:52.249529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:52.249529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:52.249529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:52.249529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:52.249529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:52.249529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:52.249529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:52.249529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.249529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:52.249914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.249914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:52.249914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:52.249914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:52.249914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.249914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:52.249914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:52.249914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:24:52.249914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:52.249914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:52.249914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.249914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:52.250611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:52.250611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:52.250611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.250611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:52.250611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:52.250611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:52.250611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:52.250611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:52.250611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:52.250611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:52.250611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:52.250611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:52.250611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:52.250611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:52.250863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:52.250863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:52.250863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.250863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:52.250863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.250863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:52.250863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:52.250863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:52.250863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:52.250863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:52.250863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.250863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:52.250863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:52.250863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:52.251160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:52.251479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:52.251479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:52.251479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:52.251479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:52.251479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:52.251479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:52.251479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:52.251479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:52.251479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:52.251479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:52.251479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.251479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:52.256706 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:52.256986 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:52.257089 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:52.257170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:52.257534 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:52.257772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.258208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:52.258208 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:52.258208 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:52.258208 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:52.258208 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:52.258208 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:52.258208 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.258208 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:52.258769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.258769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:52.258769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:52.258769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:52.258769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.258769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:52.258769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:52.258769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:52.258769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.258769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:52.258769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:52.258769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:52.258769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:52.258769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:52.258769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:52.259092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:52.259092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:52.259092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:52.259092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:52.259092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:52.259092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:52.259092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:52.259092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.259092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:52.259487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.259487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:52.259487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:52.259487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:52.259487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.259487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:52.259487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:52.259487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:52.259487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:52.259487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.259487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:52.260116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:52.260116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:52.260116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.260116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:52.260116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:52.260116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:52.260116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:52.260116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:52.260116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:52.260116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:52.260116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:52.260116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:52.260116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:52.260116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:52.260503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:52.260503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:52.260503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.260503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:52.260503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.260503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:52.260503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:52.260503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:52.260503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:52.260503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:52.260503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.260503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:52.260503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:52.260503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:52.260764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:52.261081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:52.261081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:52.261081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:52.261081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:52.261081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:52.261081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:52.261081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:52.261081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:52.261081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:52.261081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.261081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:52.265138 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:52.265409 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:52.265496 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:52.265644 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:52.266036 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:52.266233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:52.266326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.266801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:52.266801 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:52.266801 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:52.266801 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:52.266801 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:52.266801 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:52.266801 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.266801 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:52.267488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.267488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:52.267488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:52.267488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:52.267488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.267488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:52.267488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:52.267488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.267488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:52.267488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:52.267488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:52.267488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.267773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:52.267773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:52.267773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:52.267773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:52.267773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:52.267773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:52.267773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:52.267773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:52.267773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:52.267773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:52.267773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:52.267773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:52.267773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:52.267986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.267986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:52.268248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.268248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:52.268248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:52.268248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:52.268248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:52.268248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:52.268248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.268248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:52.268248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:52.268248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.268248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:52.268248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:52.268248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.268248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:52.268248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:52.268248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:52.269102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:52.269549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:52.269549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:52.269549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:56.786168 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:56.786852 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:56.786852 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:56.787168 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:56.812054 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:56.812197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:56.812678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:56.812678 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:56.812678 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:56.812678 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:56.812678 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:56.812678 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:56.812678 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:56.812678 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:56.813257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:56.813257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:56.813257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:56.813257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:56.813257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:56.813257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:56.813257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:56.813257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:56.813257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:56.813257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:56.813257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:56.813257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:56.813673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:56.813673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:56.813673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:56.813673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:56.813673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:56.813673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:56.813673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:56.813673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:56.813673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:56.813673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:56.813673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:56.813673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:56.813673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:56.814003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:56.814003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:56.814003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:56.814003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:56.814003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:56.814003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:56.814003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:56.814003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:56.814003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:56.814003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:56.814003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:56.814003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:56.814003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:56.814003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:56.814003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:56.814460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:56.814991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:56.814991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:56.814991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:56.814991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:56.814991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:56.814991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:57.240644 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:57.240972 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:57.241103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:57.241711 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:57.241885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.242365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:57.242365 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:57.242365 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:57.242365 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:57.242365 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:57.242365 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:57.242365 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.242365 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:57.243213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.243213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:57.243213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:57.243213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:57.243213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.243213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:57.243213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:57.243213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.243213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:57.243213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:57.243213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:57.243213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.243558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:57.243558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:57.243558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:57.243558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:57.243558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:57.243558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:57.243558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:57.243558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:57.243558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:57.243558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:57.243558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:57.243558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:57.243558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:57.243767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.243767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:57.244088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.244088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:57.244088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:57.244088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:57.244088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:57.244088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:57.244088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.244088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:57.244088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:57.244088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.244088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:57.244088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:57.244088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.244088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:57.244088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:57.244088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:57.244822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:57.245213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:57.245213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.245213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:57.247694 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:57.247944 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:57.248060 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:57.248165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:24:57.248715 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:57.248956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.249484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:57.249484 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:57.249484 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:57.249484 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:57.249484 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:57.249484 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:57.249484 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.249484 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:57.250369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.250369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:57.250369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:57.250369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:57.250369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.250369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:57.250369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:57.250369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:57.250369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.250369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:57.250369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:57.250369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:57.250369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:57.250369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:57.250369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:57.250768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:57.250768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:57.250768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:57.250768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:57.250768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:57.250768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:57.250768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:57.250768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.250768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:57.251230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.251230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:57.251230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:57.251230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:57.251230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.251230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:57.251230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:57.251230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:24:57.251230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:57.251230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:57.251230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.251230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:57.251559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:57.251559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:57.251559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.251559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:57.251559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:57.251559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:57.251559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:57.251559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:57.251559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:57.251559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:57.251559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:57.251559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:57.251559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:57.251559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:57.251793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:57.251793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:57.251793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.251793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:57.251950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.251950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:57.251950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:57.251950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:57.251950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:57.251950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:57.251950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.251950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:57.251950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:57.251950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.251950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:57.251950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:57.251950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.251950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:57.251950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:57.251950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:57.252565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:57.253278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:57.253278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:57.253278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:57.253278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:57.253278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:57.253278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.253278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:57.257175 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:57.257513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:57.257664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:57.257787 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:57.258224 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:57.258498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.258955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:57.258955 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:57.258955 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:57.258955 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:57.258955 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:57.258955 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:57.258955 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.258955 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:57.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:24:57.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:57.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:57.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:57.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:57.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:24:57.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:24:57.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:57.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:24:57.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:57.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:24:57.259586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:24:57.259872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:57.259872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:24:57.259872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:57.259872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:24:57.259872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:24:57.259872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:57.259872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:24:57.259872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.259872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:24:57.260195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.260195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:57.260195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:57.260195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:57.260195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.260195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:57.260195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:57.260195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:57.260195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:57.260195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.260195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:57.260480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:57.260480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:57.260480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.260480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:57.260480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:57.260480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:57.260480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:57.260480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:57.260480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:57.260480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:57.260480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:57.260480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:57.260480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:57.260480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:57.260889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:57.260889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:57.260889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.260889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.261058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:57.261639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:57.261941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:57.261941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:57.261941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.261941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:24:57.265593 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:24:57.265841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:24:57.265942 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:24:57.266066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:24:57.266477 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:57.266703 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:24:57.266806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.267235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:24:57.267235 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:57.267235 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:24:57.267235 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:24:57.267235 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:57.267235 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:24:57.267235 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.267235 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:24:57.267850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.267850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:24:57.267850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:57.267850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:24:57.267850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.267850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:57.267850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:57.267850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.267850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:57.267850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:57.267850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:24:57.267850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.268095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:24:57.268095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:57.268095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:24:57.268095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:57.268095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:24:57.268095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:24:57.268095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:57.268095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:24:57.268095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:57.268095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:24:57.268095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:24:57.268095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:57.268095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:24:57.268295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.268295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:24:57.268654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.268654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:24:57.268654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:24:57.268654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:24:57.268654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:24:57.268654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:24:57.268654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.268654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:24:57.268654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:24:57.268654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.268654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:24:57.268654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:24:57.268654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.268654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:24:57.268654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:24:57.268654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:24:57.269017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:24:57.269381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:24:57.269381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:24:57.269381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:01.786614 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:01.787368 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:01.787368 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:01.787726 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:01.816151 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:01.816436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:01.816961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:01.816961 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:01.816961 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:01.816961 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:01.816961 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:01.816961 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:01.816961 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:01.816961 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:01.817978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:01.817978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:01.817978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:01.817978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:01.817978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:01.817978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:01.817978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:01.817978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:01.817978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:01.817978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:01.817978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:01.817978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:01.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:01.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:01.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:01.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:01.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:01.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:01.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:01.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:01.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:01.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:01.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:01.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:01.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:01.820014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:01.820014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:01.820014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:01.820014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:01.820014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:01.820014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:01.820014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:01.820014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:01.820014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:01.820014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:01.820014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:01.820014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:01.820014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:01.820014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:01.820014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:01.820286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:01.820627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:01.820627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:01.820627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:01.820627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:01.820627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:01.820627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:02.241223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:02.241526 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:02.241672 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:02.242227 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:02.242495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.243112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:02.243112 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:02.243112 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:02.243112 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:02.243112 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:02.243112 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:02.243112 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.243112 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:02.243810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.243810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:02.243810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:02.243810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:02.243810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.243810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:02.243810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:02.243810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.243810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:02.243810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:02.243810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:02.243810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.244119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:02.244119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:02.244119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:02.244119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:02.244119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:02.244119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:02.244119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:02.244119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:02.244119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:02.244119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:02.244119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:02.244119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:02.244119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:02.244462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.244462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:02.244859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.244859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:02.244859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:02.244859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:02.244859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:02.244859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:02.244859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.244859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:02.244859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:02.244859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.244859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:02.244859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:02.244859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.244859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:02.244859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:02.244859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:02.245952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:02.246468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:02.246468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.246468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:02.248066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:02.248339 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:02.248473 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:02.248572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:25:02.248929 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:02.249146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.249706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:02.249706 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:02.249706 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:02.249706 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:02.249706 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:02.249706 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:02.249706 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.249706 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:02.250282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.250282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:02.250282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:02.250282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:02.250282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.250282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:02.250282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:02.250282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:02.250282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.250282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:02.250282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:02.250282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:02.250282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:02.250282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:02.250282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:02.250770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:02.250770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:02.250770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:02.250770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:02.250770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:02.250770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:02.250770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:02.250770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.250770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:02.251043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.251043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:02.251043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:02.251043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:02.251043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.251043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:02.251043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:02.251043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:25:02.251043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:02.251043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:02.251043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.251043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:02.251393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:02.251393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:02.251393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.251393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:02.251393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:02.251393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:02.251393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:02.251393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:02.251393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:02.251393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:02.251393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:02.251393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:02.251393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:02.251393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:02.251776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:02.251776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:02.251776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.251776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:02.251776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.251776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:02.251776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:02.251776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:02.251776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:02.251776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:02.251776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.251776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:02.251776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:02.251776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:02.252147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:02.252702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:02.252702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:02.252702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:02.252702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:02.252702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:02.252702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:02.252702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:02.252702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:02.252702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:02.252702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:02.252702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.252702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:02.257627 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:02.257887 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:02.258001 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:02.258074 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:02.258487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:02.258774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.259269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:02.259269 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:02.259269 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:02.259269 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:02.259269 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:02.259269 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:02.259269 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.259269 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:02.259810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.259810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:02.259810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:02.259810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:02.259810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.259810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:02.259810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:02.259810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:02.259810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.259810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:02.259810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:02.259810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:02.259810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:02.259810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:02.259810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:02.260227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:02.260227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:02.260227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:02.260227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:02.260227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:02.260227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:02.260227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:02.260227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.260227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:02.260476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.260476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:02.260476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:02.260476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:02.260476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.260476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:02.260476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:02.260476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:02.260476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:02.260476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.260476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:02.260812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:02.260812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:02.260812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.260812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:02.260812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:02.260812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:02.260812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:02.260812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:02.260812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:02.260812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:02.260812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:02.260812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:02.260812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:02.260812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:02.261178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:02.261178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:02.261178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.261178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:02.261178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.261178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:02.261178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:02.261178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:02.261178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:02.261178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:02.261178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.261178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:02.261178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:02.261178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:02.261615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:02.262103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:02.262103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:02.262103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:02.262103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:02.262103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:02.262103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:02.262103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:02.262103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:02.262103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:02.262103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.262103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:02.265971 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:02.266204 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:02.266326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:02.266435 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:02.266721 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:02.266968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:02.267032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.267465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:02.267465 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:02.267465 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:02.267465 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:02.267465 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:02.267465 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:02.267465 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.267465 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:02.268064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.268064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:02.268064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:02.268064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:02.268064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.268064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:02.268064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:02.268064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.268064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:02.268064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:02.268064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:02.268064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.268438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:02.268438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:02.268438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:02.268438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:02.268438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:02.268438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:02.268438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:02.268438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:02.268438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:02.268438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:02.268438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:02.268438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:02.268438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:02.268751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.268751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:02.268751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.268751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:02.268751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:02.268751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:02.268751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:02.268751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:02.268751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.268751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:02.268751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:02.268751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.268751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:02.268751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:02.268751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:02.269110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:02.269525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:02.269525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:02.269525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:02.269525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:02.269525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:02.269525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:06.787043 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:06.787782 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:06.787782 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:06.788320 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:06.815991 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:06.816225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:06.816801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:06.816801 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:06.816801 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:06.816801 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:06.816801 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:06.816801 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:06.816801 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:06.816801 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:06.817826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:06.817826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:06.817826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:06.817826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:06.817826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:06.817826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:06.817826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:06.817826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:06.817826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:06.817826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:06.817826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:06.817826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:06.818252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:06.818252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:06.818252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:06.818252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:06.818252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:06.818252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:06.818252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:06.818252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:06.818252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:06.818252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:06.818252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:06.818252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:06.818252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:06.818740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:06.818740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:06.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:06.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:06.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:06.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:06.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:06.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:06.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:06.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:06.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:06.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:06.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:06.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:06.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:06.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:06.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:06.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:06.819488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:06.819914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:06.819914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:06.819914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:07.241720 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:07.242084 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:07.242291 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:07.242934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:07.243167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.243816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:07.243816 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:07.243816 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:07.243816 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:07.243816 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:07.243816 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:07.243816 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.243816 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:07.244776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.244776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:07.244776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:07.244776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:07.244776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.244776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:07.244776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:07.244776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.244776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:07.244776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:07.244776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:07.244776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.245041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:07.245041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:07.245041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:07.245041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:07.245041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:07.245041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:07.245041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:07.245041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:07.245041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:07.245041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:07.245041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:07.245041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:07.245041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:07.245246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.245246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:07.245806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.245806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:07.245806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:07.245806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:07.245806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:07.245806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:07.245806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.245806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:07.245806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:07.245806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.245806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:07.245806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:07.245806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.245806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:07.245806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:07.245806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:07.246105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:07.246423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:07.246423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.246423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:07.248502 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:07.248872 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:07.249023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:07.249167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:25:07.249683 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:07.249894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.250505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:07.250505 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:07.250505 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:07.250505 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:07.250505 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:07.250505 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:07.250505 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.250505 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:07.251503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.251503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:07.251503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:07.251503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:07.251503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.251503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:07.251503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:07.251503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:07.251503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.251503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:07.251503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:07.251503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:07.251503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:07.251503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:07.251503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:07.251877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:07.251877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:07.251877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:07.251877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:07.251877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:07.251877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:07.251877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:07.251877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.251877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:07.252497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.252497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:07.252497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:07.252497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:07.252497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.252497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:07.252497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:07.252497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:25:07.252497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:07.252497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:07.252497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.252497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:07.252770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:07.252770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:07.252770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.252770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:07.252770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:07.252770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:07.252770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:07.252770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:07.252770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:07.252770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:07.252770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:07.252770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:07.252770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:07.252770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:07.253026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:07.253026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:07.253026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.253026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:07.253969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.253969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:07.253969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:07.253969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:07.253969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:07.253969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:07.253969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.253969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:07.253969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:07.253969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.253969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:07.253969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:07.253969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.253969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:07.253969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:07.253969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:07.254284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:07.254697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:07.254697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:07.254697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:07.254697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:07.254697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:07.254697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.254697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:07.258007 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:07.258393 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:07.258558 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:07.258727 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:07.259130 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:07.259395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.260021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:07.260021 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:07.260021 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:07.260021 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:07.260021 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:07.260021 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:07.260021 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.260021 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:07.260977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.260977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:07.260977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:07.260977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:07.260977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.260977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:07.260977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:07.260977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:07.260977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.260977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:07.260977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:07.260977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:07.260977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:07.260977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:07.260977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:07.261285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:07.261285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:07.261285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:07.261285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:07.261285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:07.261285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:07.261285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:07.261285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.261285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:07.262043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.262043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:07.262043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:07.262043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:07.262043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.262043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:07.262043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:07.262043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:07.262043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:07.262043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.262043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:07.262283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:07.262283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:07.262283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.262283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:07.262283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:07.262283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:07.262283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:07.262283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:07.262283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:07.262283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:07.262283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:07.262283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:07.262283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:07.262283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:07.262627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:07.262627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:07.262627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.262627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.263314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:07.263731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:07.264074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:07.264074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:07.264074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.264074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:07.266327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:07.266735 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:07.266932 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:07.267152 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:07.267570 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:07.267810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:07.267933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.268557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:07.268557 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:07.268557 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:07.268557 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:07.268557 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:07.268557 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:07.268557 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.268557 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:07.269672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.269672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:07.269672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:07.269672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:07.269672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.269672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:07.269672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:07.269672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.269672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:07.269672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:07.269672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:07.269672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.270036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:07.270036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:07.270036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:07.270036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:07.270036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:07.270036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:07.270036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:07.270036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:07.270036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:07.270036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:07.270036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:07.270036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:07.270036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:07.270374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.270374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:07.270841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.270841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:07.270841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:07.270841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:07.270841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:07.270841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:07.270841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.270841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:07.270841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:07.270841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.270841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:07.270841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:07.270841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.270841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:07.270841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:07.270841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:07.271873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:07.272861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:07.272861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:07.272861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:11.787641 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:11.788159 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:11.788159 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:11.789396 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:11.819425 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:11.819887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:11.820583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:11.820583 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:11.820583 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:11.820583 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:11.820583 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:11.820583 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:11.820583 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:11.820583 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:11.822341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:11.822341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:11.822341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:11.822341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:11.822341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:11.822341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:11.822341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:11.822341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:11.822341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:11.822341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:11.822341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:11.822341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:11.822719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:11.822719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:11.822719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:11.822719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:11.822719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:11.822719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:11.822719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:11.822719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:11.822719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:11.822719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:11.822719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:11.822719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:11.822719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:11.822913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:11.822913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:11.822913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:11.822913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:11.822913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:11.822913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:11.822913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:11.822913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:11.822913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:11.822913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:11.822913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:11.822913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:11.822913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:11.822913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:11.822913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:11.823148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:11.823437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:11.823437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:11.823437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:11.823437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:11.823437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:11.823437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:12.242233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:12.242476 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:12.242600 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:12.243035 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:12.243640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.243785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:12.243785 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:12.243785 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:12.243785 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:12.243785 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:12.243785 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:12.243785 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.243785 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:12.245177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.245177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:12.245177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:12.245177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:12.245177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.245177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:12.245177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:12.245177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.245177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:12.245177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:12.245177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:12.245177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.245449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:12.245449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:12.245449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:12.245449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:12.245449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:12.245449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:12.245449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:12.245449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:12.245449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:12.245449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:12.245449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:12.245449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:12.245449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:12.245686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.245686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:12.245686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.245686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:12.245686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:12.245686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:12.245686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:12.245686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:12.245686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.245686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:12.245686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:12.245686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.245686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:12.245686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:12.245686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:12.245945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:12.246332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:12.246332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:12.246332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:12.246332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:12.246332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.246332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:12.248990 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:12.249403 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:12.249403 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:12.249403 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:25:12.250071 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:12.250071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.250196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:12.250196 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:12.250196 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:12.250196 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:12.250196 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:12.250196 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:12.250196 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.250196 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:12.251907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.251907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:12.251907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:12.251907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:12.251907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.251907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:12.251907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:12.251907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:12.251907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.251907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:12.251907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:12.251907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:12.251907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:12.251907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:12.251907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:12.252274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:12.252274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:12.252274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:12.252274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:12.252274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:12.252274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:12.252274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:12.252274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.252274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:12.252274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.252274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:12.252274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:12.252274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:12.252274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.252697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:12.252697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:12.252697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:25:12.252697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:12.252697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:12.252697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.252697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:12.252697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:12.252697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:12.252697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.252697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:12.252697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:12.252697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:12.252913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:12.252913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:12.252913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:12.252913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:12.252913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:12.252913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:12.252913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:12.252913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:12.252913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:12.252913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:12.252913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.252913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:12.253112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.253112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:12.253112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:12.253112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:12.253112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:12.253112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:12.253112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.253112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:12.253112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:12.253112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.253112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:12.253112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:12.253112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.253112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:12.253112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:12.253112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:12.253421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:12.253993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:12.253993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:12.253993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:12.253993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:12.253993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:12.253993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.253993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:12.258445 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:12.258965 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:12.258965 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:12.258965 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:12.259907 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:12.259907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.260089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:12.260089 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:12.260089 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:12.260089 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:12.260089 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:12.260089 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:12.260089 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.260089 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:12.261899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.261899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:12.261899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:12.261899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:12.261899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.261899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:12.261899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:12.261899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:12.261899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.261899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:12.261899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:12.261899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:12.261899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:12.261899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:12.261899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:12.262620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:12.262620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:12.262620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:12.262620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:12.262620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:12.262620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:12.262620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:12.262620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.262620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:12.262620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.262620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:12.262620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:12.262620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:12.262620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.263193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:12.263193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:12.263193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:12.263193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:12.263193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.263193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:12.263193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:12.263193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:12.263193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.263193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:12.263193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:12.263193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:12.263193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:12.263742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:12.263742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:12.263742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:12.263742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:12.263742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:12.263742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:12.263742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:12.263742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:12.263742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:12.263742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.263742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:12.263742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:12.264239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.265020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:12.266754 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:12.267205 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:12.267205 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:12.267205 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:12.267464 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:12.267679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:12.267770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.269414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:12.269414 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:12.269414 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:12.269414 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:12.269414 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:12.269414 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:12.269414 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.269414 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:12.269414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.269414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:12.269414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:12.269414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:12.269414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.269414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:12.269414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:12.269725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.269725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:12.269725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:12.269725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:12.269725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.269725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:12.269725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:12.269725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:12.269725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:12.269725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:12.269725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:12.269725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:12.269725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:12.269725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:12.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:12.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:12.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:12.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:12.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:12.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:12.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:12.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:12.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:12.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:12.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:12.270216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:12.270579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:12.270579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:12.270579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:12.270579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:12.270579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:12.270579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:12.270579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:12.270579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:12.270579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:12.270579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:16.788046 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:16.788584 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:16.788584 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:16.788929 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:16.813478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:16.813640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:16.814078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:16.814078 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:16.814078 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:16.814078 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:16.814078 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:16.814078 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:16.814078 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:16.814078 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:16.814717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:16.814717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:16.814717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:16.814717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:16.814717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:16.814717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:16.814717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:16.814717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:16.814717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:16.814717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:16.814717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:16.814717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:16.814988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:16.814988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:16.814988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:16.814988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:16.814988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:16.814988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:16.814988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:16.814988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:16.814988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:16.814988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:16.814988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:16.814988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:16.814988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:16.815177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:16.815177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:16.815490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:16.815490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:16.815490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:16.815490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:16.815490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:16.815490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:16.815490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:16.815490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:16.815490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:16.815490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:16.815490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:16.815490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:16.815490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:16.815490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:16.815490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:16.815490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:16.816147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:16.816726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:16.816726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:16.816726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:17.242635 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:17.243274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:17.243274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:17.244104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:17.244104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.244219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:17.244219 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:17.244219 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:17.244219 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:17.244219 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:17.244219 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:17.244219 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.244219 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:17.245620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.245620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:17.245620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:17.245620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:17.245620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.245620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:17.245620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:17.245620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.245620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:17.245620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:17.245620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:17.245620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.245894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:17.245894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:17.245894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:17.245894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:17.245894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:17.245894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:17.245894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:17.245894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:17.245894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:17.245894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:17.245894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:17.245894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:17.245894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:17.246103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.246103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:17.246103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.246103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:17.246103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:17.246103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:17.246103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:17.246103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:17.246103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.246103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:17.246103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:17.246103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.246103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:17.246103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:17.246103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:17.246363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:17.246802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:17.246802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:17.246802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:17.246802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:17.246802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.246802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:17.249264 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:17.249565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:17.249754 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:17.249930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:25:17.250695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:17.250695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.250791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:17.250791 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:17.250791 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:17.250791 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:17.250791 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:17.250791 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:17.250791 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.250791 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:17.252723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.252723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:17.252723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:17.252723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:17.252723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.252723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:17.252723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:17.252723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:17.252723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.252723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:17.252723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:17.252723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:17.252723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:17.252723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:17.252723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:17.253359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:17.253359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:17.253359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:17.253359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:17.253359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:17.253359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:17.253359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:17.253359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.253359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:17.253359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.253359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:17.253359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:17.253359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:17.253359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.253932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:17.253932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:17.253932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:25:17.253932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:17.253932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:17.253932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.253932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:17.253932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:17.253932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:17.253932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.253932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:17.253932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:17.253932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:17.254226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:17.254226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:17.254226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:17.254226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:17.254226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:17.254226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:17.254226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:17.254226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:17.254226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:17.254226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:17.254226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.254226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:17.254425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.254425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:17.254425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:17.254425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:17.254425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:17.254425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:17.254425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.254425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:17.254425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:17.254425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.254425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:17.254425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:17.254425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.254425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:17.254425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:17.254425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:17.254704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:17.254995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:17.254995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:17.254995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:17.254995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:17.254995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:17.254995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.254995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:17.258779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:17.259242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:17.259242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:17.259242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:17.259480 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:17.259675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:17.262268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:17.262268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:17.262268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:17.262268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:17.262268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:17.262268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:17.262268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:17.262268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:17.262268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:17.262268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:17.262268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:17.262268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:17.262268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.262268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:17.262268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.262511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:17.262511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:17.262511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:17.262511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.262511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:17.262511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:17.262511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:17.262511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:17.262511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.262511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:17.262511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:17.262511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:17.262511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.262742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:17.262742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:17.262742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:17.262742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:17.262742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:17.262742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:17.262742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:17.262742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:17.262742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:17.262742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:17.262742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:17.262742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:17.262742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:17.262955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.262955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:17.262955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.262955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:17.262955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:17.262955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:17.262955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:17.262955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:17.262955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.262955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:17.262955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:17.262955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.262955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:17.262955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:17.262955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:17.263196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:17.263488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:17.263488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:17.263488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:17.263488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:17.263488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:17.263488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:17.263488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.263488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:17.267106 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:17.267519 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:17.267519 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:17.267519 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:17.267726 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:17.268236 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:17.268236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.269586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:17.269586 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:17.269586 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:17.269586 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:17.269586 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:17.269586 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:17.269586 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.269586 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:17.269586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.269586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:17.269586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:17.269586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:17.269586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.269586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:17.269586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:17.270257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.270257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:17.270257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:17.270257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:17.270257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.270257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:17.270257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:17.270257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:17.270257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:17.270257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:17.270257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:17.270257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:17.270257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:17.270257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:17.270780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:17.270780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:17.270780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:17.270780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:17.270780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.270780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:17.270780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.270780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:17.270780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:17.270780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:17.270780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:17.270780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:17.270780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.270780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:17.271312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:17.271721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:17.271721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:17.271721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:17.271721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:17.271721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:17.271721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:17.271721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:17.271721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:17.271721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:17.271721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:21.788408 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:21.789107 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:21.789107 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:21.789713 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:21.790876 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=260 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:25:21.812969 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:21.813187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:21.813729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:21.813729 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:21.813729 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:21.813729 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:21.813729 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:21.813729 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:21.813729 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:21.813729 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:21.814583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:21.814583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:21.814583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:21.814583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:21.814583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:21.814583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:21.814583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:21.814583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:21.814583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:21.814583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:21.814583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:21.814583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:21.814921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:21.814921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:21.814921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:21.814921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:21.814921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:21.814921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:21.814921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:21.814921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:21.814921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:21.814921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:21.814921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:21.814921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:21.814921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:21.815259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:21.815259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:21.815689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:21.815689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:21.815689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:21.815689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:21.815689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:21.815689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:21.815689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:21.815689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:21.815689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:21.815689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:21.815689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:21.815689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:21.815689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:21.815689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:21.815689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:21.815689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:21.815975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:21.816253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:21.816253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:21.816253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:22.243037 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:22.243221 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:22.243319 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:22.243977 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:22.244153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.244606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:22.244606 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:22.244606 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:22.244606 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:22.244606 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:22.244606 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:22.244606 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.244606 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:22.245241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.245241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:22.245241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:22.245241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:22.245241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.245241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:22.245241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:22.245241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.245241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:22.245241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:22.245241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:22.245241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.245491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:22.245491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:22.245491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:22.245491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:22.245491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:22.245491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:22.245491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:22.245491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:22.245491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:22.245491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:22.245491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:22.245491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:22.245491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:22.245788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.245788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:22.246015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.246015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:22.246015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:22.246015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:22.246015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:22.246015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:22.246015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.246015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:22.246015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:22.246015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.246015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:22.246015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:22.246015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.246015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:22.246015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:22.246015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:22.246652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:22.247074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:22.247074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.247074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:22.249608 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:22.249790 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:22.249868 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:22.249965 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:25:22.250287 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:22.250463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.250845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:22.250845 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:22.250845 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:22.250845 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:22.250845 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:22.250845 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:22.250845 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.250845 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:22.251355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.251355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:22.251355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:22.251355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:22.251355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.251355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:22.251355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:22.251355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:22.251355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.251355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:22.251355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:22.251355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:22.251355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:22.251355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:22.251355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:22.251734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:22.251734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:22.251734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:22.251734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:22.251734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:22.251734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:22.251734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:22.251734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.251734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:22.251884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.251884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:22.251884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:22.251884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:22.251884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.251884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:22.251884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:22.251884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:25:22.251884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:22.251884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:22.251884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.251884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:22.252100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:22.252100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:22.252100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.252100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:22.252100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:22.252100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:22.252100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:22.252100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:22.252100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:22.252100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:22.252100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:22.252100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:22.252100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:22.252100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:22.252300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:22.252300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:22.252300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.252300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:22.252634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.252634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:22.252634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:22.252634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:22.252634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:22.252634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:22.252634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.252634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:22.252634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:22.252634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.252634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:22.252634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:22.252634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.252634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:22.252634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:22.252634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:22.253068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:22.253743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:22.253743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:22.253743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:22.253743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:22.253743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:22.253743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.253743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:22.259039 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:22.259254 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:22.259349 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:22.259446 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:22.259759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:22.259925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.260294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:22.260294 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:22.260294 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:22.260294 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:22.260294 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:22.260294 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:22.260294 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.260294 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:22.260837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.260837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:22.260837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:22.260837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:22.260837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.260837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:22.260837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:22.260837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:22.260837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.260837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:22.260837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:22.260837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:22.260837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:22.260837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:22.260837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:22.261304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:22.261304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:22.261304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:22.261304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:22.261304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:22.261304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:22.261304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:22.261304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.261304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:22.261482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.261482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:22.261482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:22.261482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:22.261482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.261482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:22.261482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:22.261482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:22.261482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:22.261482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.261482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:22.261678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:22.261678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:22.261678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.261678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:22.261678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:22.261678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:22.261678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:22.261678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:22.261678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:22.261678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:22.261678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:22.261678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:22.261678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:22.261678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:22.261942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:22.261942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:22.261942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.261942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.262305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:22.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:22.263650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:22.263650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:22.263650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.263650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:22.267332 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:22.267520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:22.267619 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:22.267747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:22.267994 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:22.268153 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:22.268235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.268604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:22.268604 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:22.268604 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:22.268604 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:22.268604 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:22.268604 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:22.268604 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.268604 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:22.269138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.269138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:22.269138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:22.269138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:22.269138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.269138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:22.269138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:22.269138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.269138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:22.269138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:22.269138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:22.269138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.269377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:22.269377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:22.269377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:22.269377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:22.269377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:22.269377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:22.269377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:22.269377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:22.269377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:22.269377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:22.269377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:22.269377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:22.269377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:22.269612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.269612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:22.269766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.269766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:22.269766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:22.269766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:22.269766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:22.269766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:22.269766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.269766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:22.269766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:22.269766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.269766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:22.269766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:22.269766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.269766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:22.269766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:22.269766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:22.270302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:22.270899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:22.270899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:22.270899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:26.788905 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:26.789371 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:26.789591 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:26.790428 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:26.814066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:26.814255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:26.814800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:26.814800 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:26.814800 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:26.814800 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:26.814800 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:26.814800 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:26.814800 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:26.814800 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:26.815485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:26.815485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:26.815485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:26.815485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:26.815485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:26.815485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:26.815485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:26.815485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:26.815485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:26.815485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:26.815485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:26.815485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:26.815726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:26.815726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:26.815726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:26.815726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:26.815726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:26.815726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:26.815726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:26.815726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:26.815726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:26.815726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:26.815726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:26.815726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:26.815726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:26.816042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:26.816042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:26.816146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:26.816146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:26.816146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:26.816146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:26.816146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:26.816146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:26.816146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:26.816146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:26.816146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:26.816146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:26.816146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:26.816146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:26.816146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:26.816146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:26.816146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:26.816146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:26.816388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:26.816710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:26.816710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:26.816710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:27.243446 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:27.243783 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:27.243858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:27.244498 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:27.244697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.245134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:27.245134 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:27.245134 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:27.245134 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:27.245134 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:27.245134 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:27.245134 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.245134 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:27.245856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.245856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:27.245856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:27.245856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:27.245856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.245856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:27.245856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:27.245856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.245856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:27.245856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:27.245856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:27.245856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.246468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:27.246468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:27.246468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:27.246468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:27.246468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:27.246468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:27.246468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:27.246468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:27.246468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:27.246468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:27.246468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:27.246468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:27.246468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:27.246930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.246930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:27.246930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.246930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:27.246930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:27.246930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:27.246930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:27.246930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:27.246930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.246930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:27.246930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:27.246930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.246930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:27.246930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:27.246930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:27.248122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:27.249011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:27.249011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:27.249011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:27.249011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:27.249011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.249011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:27.249959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:27.250205 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:27.250291 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:27.250398 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:25:27.250717 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:27.250860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.251233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:27.251233 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:27.251233 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:27.251233 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:27.251233 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:27.251233 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:27.251233 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.251233 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:27.251826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.251826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:27.251826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:27.251826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:27.251826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.251826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:27.251826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:27.251826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:27.251826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.251826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:27.251826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:27.251826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:27.251826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:27.251826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:27.251826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:27.252121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:27.252121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:27.252121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:27.252121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:27.252121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:27.252121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:27.252121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:27.252121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.252121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:27.252424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.252424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:27.252424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:27.252424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:27.252424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.252424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:27.252424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:27.252424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:25:27.252424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:27.252424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:27.252424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.252424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:27.252914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:27.252914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:27.252914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.252914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:27.252914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:27.252914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:27.252914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:27.252914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:27.252914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:27.252914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:27.252914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:27.252914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:27.252914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:27.252914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:27.253382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:27.253382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:27.253382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.253382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:27.253382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.253382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:27.253382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:27.253382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:27.253382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:27.253382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:27.253382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.253382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:27.253382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:27.253382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:27.253687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:27.254025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:27.254025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:27.254025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:27.254025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:27.254025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:27.254025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:27.254025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:27.254025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:27.254025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:27.254025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:27.254025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.254025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:27.259370 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:27.259799 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:27.259903 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:27.260015 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:27.260365 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:27.260552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.260952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:27.260952 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:27.260952 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:27.260952 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:27.260952 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:27.260952 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:27.260952 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.260952 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:27.261722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.261722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:27.261722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:27.261722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:27.261722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.261722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:27.261722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:27.261722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:27.261722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.261722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:27.261722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:27.261722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:27.261722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:27.261722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:27.261722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:27.262010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:27.262010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:27.262010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:27.262010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:27.262010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:27.262010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:27.262010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:27.262010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.262010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:27.262330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.262330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:27.262330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:27.262330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:27.262330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.262330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:27.262330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:27.262330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:27.262330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:27.262330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.262330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:27.262807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:27.262807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:27.262807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.262807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:27.262807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:27.262807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:27.262807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:27.262807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:27.262807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:27.262807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:27.262807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:27.262807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:27.262807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:27.262807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:27.263281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:27.263281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:27.263281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.263281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:27.263281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.263281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:27.263281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:27.263281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:27.263281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:27.263281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:27.263281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.263281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:27.263281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:27.263281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:27.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:27.263938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:27.263938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:27.263938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:27.263938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:27.263938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:27.263938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:27.263938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:27.263938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:27.263938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:27.263938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.263938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:27.267733 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:27.268000 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:27.268087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:27.268223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:27.268526 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:27.268713 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:27.268795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.269163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:27.269163 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:27.269163 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:27.269163 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:27.269163 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:27.269163 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:27.269163 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.269163 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:27.269837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.269837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:27.269837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:27.269837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:27.269837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.269837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:27.269837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:27.269837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.269837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:27.269837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:27.269837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:27.269837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.270067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:27.270067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:27.270067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:27.270067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:27.270067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:27.270067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:27.270067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:27.270067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:27.270067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:27.270067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:27.270067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:27.270067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:27.270067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:27.270264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.270264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:27.270498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.270498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:27.270498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:27.270498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:27.270498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:27.270498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:27.270498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.270498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:27.270498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:27.270498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.270498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:27.270498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:27.270498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.270498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:27.270498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:27.270498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:27.270816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:27.271111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:27.271111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:27.271111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:31.789339 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:31.789860 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:31.789903 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:31.790654 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:31.814851 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:31.815005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:31.815462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:31.815462 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:31.815462 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:31.815462 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:31.815462 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:31.815462 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:31.815462 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:31.815462 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:31.816212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:31.816212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:31.816212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:31.816212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:31.816212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:31.816212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:31.816212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:31.816212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:31.816212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:31.816212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:31.816212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:31.816212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:31.816613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:31.816613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:31.816613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:31.816613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:31.816613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:31.816613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:31.816613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:31.816613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:31.816613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:31.816613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:31.816613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:31.816613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:31.816613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:31.816927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:31.816927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:31.817257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:31.817257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:31.817257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:31.817257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:31.817257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:31.817257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:31.817257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:31.817257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:31.817257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:31.817257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:31.817257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:31.817257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:31.817257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:31.817257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:31.817257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:31.817257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:31.818218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:31.818909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:31.818909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:31.818909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:32.243923 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:32.244297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:32.244556 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:32.245181 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:32.245331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.245807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:32.245807 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:32.245807 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:32.245807 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:32.245807 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:32.245807 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:32.245807 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.245807 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:32.246579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.246579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:32.246579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:32.246579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:32.246579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.246579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:32.246579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:32.246579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.246579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:32.246579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:32.246579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:32.246579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.246955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:32.246955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:32.246955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:32.246955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:32.246955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:32.246955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:32.246955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:32.246955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:32.246955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:32.246955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:32.246955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:32.246955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:32.246955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:32.247244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.247244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:32.247600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.247600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:32.247600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:32.247600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:32.247600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:32.247600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:32.247600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.247600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:32.247600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:32.247600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.247600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:32.247600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:32.247600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.247600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:32.247600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:32.247600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:32.248169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:32.249002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:32.249002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.249002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:32.250360 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:32.250725 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:32.250840 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:32.250960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:25:32.251276 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:32.251437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.251795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:32.251795 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:32.251795 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:32.251795 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:32.251795 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:32.251795 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:32.251795 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.251795 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:32.252633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.252633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:32.252633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:32.252633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:32.252633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.252633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:32.252633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:32.252633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:32.252633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.252633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:32.252633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:32.252633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:32.252633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:32.252633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:32.252633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:32.253018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:32.253018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:32.253018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:32.253018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:32.253018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:32.253018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:32.253018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:32.253018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.253018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:32.253482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.253482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:32.253482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:32.253482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:32.253482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.253482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:32.253482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:32.253482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:25:32.253482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:32.253482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:32.253482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.253482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:32.254075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:32.254075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:32.254075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.254075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:32.254075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:32.254075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:32.254075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:32.254075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:32.254075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:32.254075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:32.254075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:32.254075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:32.254075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:32.254075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:32.254765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:32.254765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:32.254765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.254765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:32.254765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.254765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:32.254765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:32.254765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:32.254765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:32.254765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:32.254765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.254765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:32.254765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:32.254765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:32.255184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:32.255629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:32.255629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:32.255629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:32.255629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:32.255629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:32.255629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:32.255629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:32.255629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:32.255629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:32.255629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:32.255629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.255629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:32.259770 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:32.260026 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:32.260137 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:32.260278 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:32.260638 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:32.260818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.261202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:32.261202 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:32.261202 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:32.261202 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:32.261202 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:32.261202 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:32.261202 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.261202 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:32.262137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.262137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:32.262137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:32.262137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:32.262137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.262137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:32.262137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:32.262137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:32.262137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.262137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:32.262137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:32.262137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:32.262137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:32.262137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:32.262137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:32.262631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:32.262631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:32.262631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:32.262631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:32.262631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:32.262631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:32.262631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:32.262631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.262631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:32.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:32.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:32.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:32.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:32.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:32.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:32.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:32.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:32.263831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:32.263831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:32.263831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.263831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:32.263831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:32.263831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:32.263831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:32.263831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:32.263831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:32.263831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:32.263831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:32.263831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:32.263831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:32.263831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:32.264305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:32.264305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:32.264305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.264305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:32.264305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.264305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:32.264305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:32.264305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:32.264305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:32.264305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:32.264305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.264305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:32.264305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:32.264305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:32.264682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:32.265103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:32.265103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:32.265103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:32.265103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:32.265103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:32.265103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:32.265103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:32.265103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:32.265103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:32.265103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.265103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:32.268172 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:32.268511 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:32.268643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:32.268821 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:32.269098 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:32.269261 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:32.269341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.269752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:32.269752 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:32.269752 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:32.269752 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:32.269752 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:32.269752 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:32.269752 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.269752 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:32.270452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.270452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:32.270452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:32.270452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:32.270452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.270452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:32.270452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:32.270452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.270452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:32.270452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:32.270452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:32.270452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.270811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:32.270811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:32.270811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:32.270811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:32.270811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:32.270811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:32.270811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:32.270811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:32.270811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:32.270811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:32.270811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:32.270811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:32.270811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:32.271139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.271139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:32.271198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.271198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:32.271198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:32.271198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:32.271198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:32.271198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:32.271198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.271198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:32.271198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:32.271198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.271198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:32.271198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:32.271198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.271198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:32.271198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:32.271198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:32.271605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:32.272062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:32.272062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:32.272062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:36.789821 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:36.790359 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:36.790359 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:36.791014 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:36.816303 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:36.816455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:36.816883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:36.816883 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:36.816883 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:36.816883 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:36.816883 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:36.816883 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:36.816883 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:36.816883 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:36.817466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:36.817466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:36.817466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:36.817466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:36.817466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:36.817466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:36.817466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:36.817466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:36.817466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:36.817466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:36.817466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:36.817466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:36.817730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:36.817730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:36.817730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:36.817730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:36.817730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:36.817730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:36.817730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:36.817730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:36.817730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:36.817730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:36.817730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:36.817730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:36.817730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:36.817963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:36.817963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:36.818141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:36.818141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:36.818141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:36.818141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:36.818141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:36.818141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:36.818141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:36.818141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:36.818141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:36.818141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:36.818141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:36.818141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:36.818141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:36.818141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:36.818141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:36.818141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:36.818441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:36.818765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:36.818765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:36.818765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:37.244328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:37.244769 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:37.244889 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:37.245387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:37.245550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.246061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:37.246061 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:37.246061 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:37.246061 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:37.246061 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:37.246061 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:37.246061 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.246061 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:37.246664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.246664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:37.246664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:37.246664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:37.246664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.246664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:37.246664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:37.246664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.246664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:37.246664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:37.246664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:37.246664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.246902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:37.246902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:37.246902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:37.246902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:37.246902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:37.246902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:37.246902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:37.246902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:37.246902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:37.246902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:37.246902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:37.246902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:37.246902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:37.247125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.247125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:37.247369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.247369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:37.247369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:37.247369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:37.247369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:37.247369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:37.247369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.247369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:37.247369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:37.247369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.247369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:37.247369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:37.247369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.247369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:37.247369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:37.247369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:37.247659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:37.247987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:37.247987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.247987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:37.250824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:37.251098 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:37.251282 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:37.251282 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:25:37.251636 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:37.251971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.252246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:37.252246 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:37.252246 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:37.252246 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:37.252246 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:37.252246 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:37.252246 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.252246 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:37.252801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.252801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:37.252801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:37.252801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:37.252801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.252801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:37.252801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:37.252801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:37.252801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.252801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:37.252801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:37.252801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:37.252801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:37.252801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:37.252801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:37.253208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:37.253208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:37.253208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:37.253208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:37.253208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:37.253208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:37.253208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:37.253208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.253208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:37.253453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.253453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:37.253453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:37.253453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:37.253453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.253453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:37.253453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:37.253453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:25:37.253453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:37.253453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:37.253453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.253453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:37.253793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:37.253793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:37.253793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.253793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:37.253793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:37.253793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:37.253793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:37.253793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:37.253793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:37.253793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:37.253793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:37.253793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:37.253793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:37.253793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:37.254131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:37.254131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:37.254131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.254131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:37.254131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.254131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:37.254131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:37.254131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:37.254131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:37.254131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:37.254131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.254131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:37.254131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:37.254131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:37.254529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:37.255037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:37.255037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:37.255037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:37.255037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:37.255037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:37.255037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:37.255037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:37.255037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:37.255037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:37.255037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:37.255037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.255037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:37.260155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:37.260460 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:37.260620 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:37.260761 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:37.261127 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:37.261381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.261848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:37.261848 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:37.261848 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:37.261848 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:37.261848 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:37.261848 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:37.261848 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.261848 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:37.262597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.262597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:37.262597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:37.262597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:37.262597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.262597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:37.262597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:37.262597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:37.262597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.262597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:37.262597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:37.262597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:37.262597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:37.262597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:37.262597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:37.263006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:37.263006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:37.263006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:37.263006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:37.263006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:37.263006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:37.263006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:37.263006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.263006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:37.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:37.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:37.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:37.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:37.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:37.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:37.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:37.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:37.263820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:37.263820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:37.263820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.263820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:37.263820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:37.263820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:37.263820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:37.263820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:37.263820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:37.263820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:37.263820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:37.263820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:37.263820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:37.263820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:37.264186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:37.264186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:37.264186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.264186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.264513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:37.264996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:37.265488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:37.265488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:37.265488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.265488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:37.268611 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:37.268900 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:37.269045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:37.269227 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:37.269629 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:37.269897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:37.270163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.270786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:37.270786 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:37.270786 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:37.270786 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:37.270786 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:37.270786 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:37.270786 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.270786 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:37.271687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.271687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:37.271687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:37.271687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:37.271687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.271687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:37.271687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:37.271687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.271687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:37.271687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:37.271687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:37.271687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.272074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:37.272074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:37.272074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:37.272074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:37.272074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:37.272074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:37.272074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:37.272074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:37.272074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:37.272074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:37.272074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:37.272074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:37.272074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:37.272381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.272381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:37.272626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.272626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:37.272626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:37.272626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:37.272626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:37.272626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:37.272626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.272626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:37.272626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:37.272626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.272626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:37.272626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:37.272626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.272626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:37.272626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:37.272626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:37.272964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:37.273334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:37.273334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:37.273334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:41.790573 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:41.794675 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:41.794675 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:41.794675 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:41.822935 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:41.823129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:41.823506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:41.823506 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:41.823506 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:41.823506 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:41.823506 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:41.823506 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:41.823506 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:41.823506 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:41.824146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:41.824146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:41.824146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:41.824146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:41.824146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:41.824146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:41.824146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:41.824146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:41.824146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:41.824146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:41.824146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:41.824146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:41.824907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:41.824907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:41.824907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:41.824907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:41.824907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:41.824907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:41.824907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:41.824907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:41.824907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:41.824907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:41.824907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:41.824907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:41.824907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:41.825484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:41.825484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:41.825484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:41.825484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:41.825484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:41.825484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:41.825484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:41.825484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:41.825484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:41.825484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:41.825484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:41.825484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:41.825484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:41.825484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:41.825484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:41.826716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:41.827037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:41.827037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:41.827037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:41.827037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:41.827037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:41.827037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:42.244893 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:42.245809 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:42.245809 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:42.245914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:42.246061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.246498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:42.246498 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:42.246498 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:42.246498 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:42.246498 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:42.246498 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:42.246498 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.246498 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:42.249029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.249029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:42.249029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:42.249029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:42.249029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.249029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:42.249029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:42.249029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.249029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:42.249029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:42.249029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:42.249029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.249431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:42.249431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:42.249431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:42.249431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:42.249431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:42.249431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:42.249431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:42.249431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:42.249431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:42.249431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:42.249431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:42.249431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:42.249431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:42.249802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.249802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:42.249802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.249802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:42.249802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:42.249802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:42.249802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:42.249802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:42.249802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.249802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:42.249802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:42.249802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.249802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:42.249802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:42.249802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:42.250182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:42.250646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:42.250646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:42.250646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:42.250646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:42.250646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.250646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:42.251900 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:42.251900 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:42.251900 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:42.251900 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:25:42.252193 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:42.252254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.255220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:42.255747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:42.255747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:42.255747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:42.255747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:42.255747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:42.255747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:42.255747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:42.255747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:42.255747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:42.255747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:42.255747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:42.255747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:42.255747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.255747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:42.255747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:42.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:42.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:42.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:42.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:42.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:25:42.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:42.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:42.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:42.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:42.256150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:42.256502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.256502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:42.256502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:42.256502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:42.256502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:42.256502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:42.256502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:42.256502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:42.256502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:42.256502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:42.256502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:42.256502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:42.256502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:42.256502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:42.256878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.256878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:42.256878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.256878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:42.256878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:42.256878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:42.256878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:42.256878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:42.256878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.256878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:42.256878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:42.256878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.256878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:42.256878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:42.256878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:42.257251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:42.257716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:42.257716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:42.257716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:42.257716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:42.257716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:42.257716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:42.257716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:42.257716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.257716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:42.260483 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:42.261134 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:42.261134 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:42.261134 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:42.261335 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:42.261523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.264556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:42.265001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:42.265001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:42.265001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:42.265001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:42.265001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:42.265001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:42.265001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:42.265001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:42.265001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:42.265001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:42.265001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:42.265001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:42.265001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.265001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:42.265001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.265419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:42.265419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:42.265419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:42.265419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.265419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:42.265419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:42.265419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:42.265419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:42.265419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.265419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:42.265419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:42.265419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:42.265419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.265763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:42.265763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:42.265763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:42.265763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:42.265763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:42.265763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:42.265763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:42.265763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:42.265763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:42.265763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:42.265763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:42.265763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:42.265763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:42.266066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.266066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:42.266066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.266066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:42.266066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:42.266066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:42.266066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:42.266066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:42.266066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.266066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:42.266066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:42.266066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.266066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:42.266066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:42.266066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:42.266437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:42.266900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:42.266900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:42.266900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:42.266900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:42.266900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:42.266900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:42.266900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.266900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:42.268983 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:42.269638 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:42.269638 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:42.269638 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:42.269852 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:42.270076 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:42.270150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.271571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:42.271571 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:42.271571 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:42.271571 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:42.271571 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:42.271571 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:42.271571 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.271571 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:42.271571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.271571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:42.271571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:42.271571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:42.271571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.271571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:42.271571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:42.271977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.271977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:42.271977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:42.271977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:42.271977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.271977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:42.271977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:42.271977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:42.271977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:42.271977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:42.271977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:42.271977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:42.271977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:42.271977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:42.272309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:42.272309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:42.272309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:42.272309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:42.272309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.272309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:42.273562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.273562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:42.273562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:42.273562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:42.273562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:42.273562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:42.273562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.273562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:42.273562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:42.273562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.273562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:42.273562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:42.273562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.273562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:42.273562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:42.273562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:42.274427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:42.275360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:42.275360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:42.275360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:46.790699 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:46.792680 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:46.792680 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:46.792680 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:46.839662 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:46.839954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:46.840468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:46.840468 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:46.840468 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:46.840468 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:46.840468 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:46.840468 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:46.840468 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:46.840468 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:46.841195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.049 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:46.841195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:46.841195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:46.841195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:46.841195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:46.841195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:46.841195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:46.841195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:46.841195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:46.841195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:46.841195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:46.841195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:46.841589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:46.841589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:46.841589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:46.841589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:46.841589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:46.841589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:46.841589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:46.841589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:46.841589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:46.841589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:46.841589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:46.841589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:46.841589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:46.841939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:46.841939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:46.842004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:46.842004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:46.842004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:46.842004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:46.842004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:46.842004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:46.842004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:46.842004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:46.842004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:46.842004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:46.842004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:46.842004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:46.842004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:46.842004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:46.842004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:46.842004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:46.842343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:46.842851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:46.842851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:46.842851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:47.245635 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:47.246162 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:47.246275 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:47.246945 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:47.247201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.247880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:47.247880 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:47.247880 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:47.247880 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:47.247880 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:47.247880 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:47.247880 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.247880 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:47.249175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.249175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:47.249175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:47.249175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:47.249175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.249175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:47.249175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:47.249175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.249175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:47.249175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:47.249175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:47.249175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.249510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:47.249510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:47.249510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:47.249510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:47.249510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:47.249510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:47.249510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:47.249510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:47.249510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:47.249510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:47.249510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:47.249510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:47.249510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:47.249742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.249742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:47.250801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.250801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:47.250801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:47.250801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:47.250801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:47.250801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:47.250801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.250801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:47.250801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:47.250801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.250801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:47.250801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:47.250801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.250801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:47.250801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:47.250801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:47.251102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:47.251575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:47.251575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.251575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:47.251860 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:47.252117 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:47.252250 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:47.252419 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:25:47.252884 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:47.253149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.253709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:47.253709 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:47.253709 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:47.253709 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:47.253709 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:47.253709 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:47.253709 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.253709 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:47.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:47.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:47.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:47.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:47.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:47.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:47.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:47.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:47.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:47.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:47.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:47.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:47.255024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:47.255024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:47.255024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:47.255024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:47.255024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:47.255024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:47.255024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:47.255024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.255024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:47.256050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.256050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:47.256050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:47.256050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:47.256050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.256050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:47.256050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:47.256050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:25:47.256050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:47.256050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:47.256050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.256050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:47.256354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:47.256354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:47.256354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.256354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:47.256354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:47.256354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:47.256354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:47.256354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:47.256354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:47.256354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:47.256354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:47.256354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:47.256354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:47.256354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:47.256643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:47.256643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:47.256643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.256643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:47.257902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.257902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:47.257902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:47.257902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:47.257902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:47.257902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:47.257902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.257902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:47.257902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:47.257902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.257902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:47.257902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:47.257902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.257902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:47.257902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:47.257902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:47.258504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:47.259180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:47.259180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:47.259180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:47.259180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:47.259180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:47.259180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.259180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:47.260898 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:47.261310 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:47.261469 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:47.261872 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:47.262300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:47.262665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.263275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:47.263275 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:47.263275 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:47.263275 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:47.263275 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:47.263275 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:47.263275 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.263275 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:47.264394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.264394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:47.264394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:47.264394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:47.264394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.264394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:47.264394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:47.264394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:47.264394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.264394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:47.264394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:47.264394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:47.264394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:47.264394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:47.264394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:47.264718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:47.264718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:47.264718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:47.264718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:47.264718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:47.264718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:47.264718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:47.264718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.264718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:47.266010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.266010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:47.266010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:47.266010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:47.266010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.266010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:47.266010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:47.266010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:47.266010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:47.266010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.266010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:47.266264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:47.266264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:47.266264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.266264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:47.266264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:47.266264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:47.266264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:47.266264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:47.266264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:47.266264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:47.266264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:47.266264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:47.266264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:47.266264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:47.266522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:47.266522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:47.266522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.266522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.267831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:47.268463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:47.268794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:47.268794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:47.268794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.268794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:47.269415 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:47.269771 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:47.269913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:47.270119 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:47.270506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:47.270806 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:47.270973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:47.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:47.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:47.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:47.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:47.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:47.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:47.272677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.272677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:47.272677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:47.272677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:47.272677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.272677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:47.272677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:47.272677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.272677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:47.272677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:47.272677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:47.272677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.272926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:47.272926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:47.272926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:47.272926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:47.272926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:47.272926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:47.272926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:47.272926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:47.272926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:47.272926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:47.272926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:47.272926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:47.272926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:47.273128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.273128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:47.274246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.274246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:47.274246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:47.274246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:47.274246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:47.274246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:47.274246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.274246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:47.274246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:47.274246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.274246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:47.274246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:47.274246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.274246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:47.274246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:47.274246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:47.274532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:47.274839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:47.274839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:47.274839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:51.791574 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:51.792179 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:51.792551 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:51.793056 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:51.822664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:51.822916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:51.823522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:51.823522 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:51.823522 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:51.823522 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:51.823522 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:51.823522 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:51.823522 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:51.823522 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:51.824320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:51.824320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:51.824320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:51.824320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:51.824320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:51.824320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:51.824320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:51.824320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:51.824320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:51.824320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:51.824320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:51.824320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:51.825600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:51.825600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:51.825600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:51.825600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:51.825600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:51.825600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:51.825600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:51.825600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:51.825600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:51.825600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:51.825600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:51.825600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:51.825600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:51.826356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:51.826356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:51.826356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:51.826356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:51.826356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:51.826356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:51.826356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:51.826356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:51.826356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:51.826356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:51.826356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:51.826356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:51.826356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:51.826356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:51.826356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:51.826796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:51.827337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:51.827337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:51.827337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:51.827337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:51.827337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:51.827337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:52.246136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:52.246787 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:52.246919 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:52.247429 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:52.247622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.248165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:52.248165 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:52.248165 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:52.248165 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:52.248165 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:52.248165 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:52.248165 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.248165 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:52.249101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.249101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:52.249101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:52.249101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:52.249101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.249101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:52.249101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:52.249101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.249101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:52.249101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:52.249101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:52.249101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.249491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:52.249491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:52.249491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:52.249491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:52.249491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:52.249491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:52.249491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:52.249491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:52.249491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:52.249491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:52.249491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:52.249491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:52.249491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:52.249823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.249823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:52.250146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.250146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:52.250146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:52.250146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:52.250146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:52.250146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:52.250146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.250146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:52.250146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:52.250146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.250146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:52.250146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:52.250146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.250146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:52.250146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:52.250146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:52.250553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:52.250886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:52.250886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.250886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:52.252251 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:52.252646 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:52.252837 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:52.252837 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:25:52.253182 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:52.253371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.253988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:52.253988 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:52.253988 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:52.253988 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:52.253988 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:52.253988 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:52.253988 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.253988 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:52.254623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.254623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:52.254623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:52.254623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:52.254623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.254623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:52.254623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:52.254623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:52.254623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.254623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:52.254623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:52.254623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:52.254623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:52.254623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:52.254623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:52.255031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:52.255031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:52.255031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:52.255031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:52.255031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:52.255031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:52.255031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:52.255031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.255031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:52.255443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.255443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:52.255443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:52.255443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:52.255443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.255443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:52.255443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:52.255443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:25:52.255443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:52.255443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:52.255443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.255443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:52.255945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:52.255945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:52.255945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.255945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:52.255945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:52.255945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:52.255945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:52.255945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:52.255945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:52.255945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:52.255945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:52.255945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:52.255945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:52.255945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:52.256368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:52.256368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:52.256368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.256368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:52.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:52.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:52.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:52.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:52.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:52.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:52.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:52.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:52.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:52.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:52.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:52.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:52.256984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:52.257796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:52.257796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:52.257796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:52.257796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:52.257796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:52.257796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.257796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:52.261423 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:52.261850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:52.261959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:52.262091 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:52.262420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:52.262689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.263202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:52.263202 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:52.263202 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:52.263202 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:52.263202 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:52.263202 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:52.263202 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.263202 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:52.263945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.263945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:52.263945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:52.263945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:52.263945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.263945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:52.263945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:52.263945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:52.263945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.263945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:52.263945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:52.263945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:52.263945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:52.263945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:52.263945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:52.264510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:52.264510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:52.264510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:52.264510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:52.264510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:52.264510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:52.264510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:52.264510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.264510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:52.264935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.264935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:52.264935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:52.264935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:52.264935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.264935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:52.264935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:52.264935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:52.264935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:52.264935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.264935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:52.265283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:52.265283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:52.265283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.265283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:52.265283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:52.265283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:52.265283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:52.265283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:52.265283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:52.265283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:52.265283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:52.265283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:52.265283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:52.265283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:52.265732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:52.265732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:52.265732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.265732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.266050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:52.266667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:52.267147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:52.267147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:52.267147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.267147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:52.269942 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:52.270329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:52.270533 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:52.270671 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:52.270982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:52.271221 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:52.271329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.271882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:52.271882 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:52.271882 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:52.271882 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:52.271882 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:52.271882 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:52.271882 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.271882 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:52.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:52.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:52.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:52.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:52.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:52.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:52.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:52.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:52.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.273628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:52.273628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:52.273628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:52.273628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:52.273628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:52.273628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:52.273628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:52.273628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:52.273628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:52.273628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:52.273628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:52.273628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:52.273628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:52.273951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.273951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:52.274218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.274218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:52.274218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:52.274218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:52.274218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:52.274218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:52.274218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.274218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:52.274218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:52.274218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.274218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:52.274218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:52.274218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.274218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:52.274218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:52.274218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:52.274690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:52.275235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:52.275235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:52.275235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:56.792173 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:56.794975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:56.794975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:56.794975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:56.840135 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:56.840396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:56.842878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:56.842878 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:56.842878 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:56.842878 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:56.842878 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:56.842878 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:56.842878 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:56.842878 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:56.842878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:56.842878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:56.842878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:56.842878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:56.842878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:56.842878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:56.842878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:56.843200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:56.843200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:56.843200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:56.843200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:56.843200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:56.843200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:56.843200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:56.843200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:56.843200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:56.843200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:56.843200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:56.843200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:56.843200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:56.843200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:56.843572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:56.843572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:56.843572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:56.843572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:56.843572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:56.843572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:56.843572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:56.843572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:56.843572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:56.843572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:56.843572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:56.843572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:56.843572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:56.843572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:56.843920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:56.844311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:56.844311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:56.844311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:56.844311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:56.844311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:56.844311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:56.844311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:56.844311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:56.844311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:57.246826 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:57.247640 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:57.247640 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:57.248059 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:57.248249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.248869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:57.248869 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:57.248869 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:57.248869 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:57.248869 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:57.248869 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:57.248869 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.248869 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:57.249791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.249791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:57.249791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:57.249791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:57.249791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.249791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:57.249791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:57.249791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.249791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:57.249791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:57.249791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:57.249791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.250263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:57.250263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:57.250263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:57.250263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:57.250263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:57.250263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:57.250263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:57.250263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:57.250263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:57.250263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:57.250263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:57.250263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:57.250263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:57.250794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.250794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:57.250893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.250893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:57.250893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:57.250893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:57.250893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:57.250893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:57.250893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.250893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:57.250893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:57.250893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.250893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:57.250893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:57.250893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.250893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:57.250893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:57.250893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:57.251263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:57.251689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:57.251689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.251689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:57.252766 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:57.253258 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:57.253382 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:57.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:25:57.254066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:57.254298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.254975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:57.254975 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:57.254975 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:57.254975 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:57.254975 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:57.254975 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:57.254975 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.254975 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:57.255924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.255924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:57.255924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:57.255924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:57.255924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.255924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:57.255924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:57.255924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:57.255924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.255924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:57.255924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:57.255924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:57.255924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:57.255924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:57.255924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:57.256326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:57.256326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:57.256326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:57.256326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:57.256326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:57.256326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:57.256326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:57.256326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.256326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:57.256791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.256791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:57.256791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:57.256791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:57.256791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.256791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:57.256791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:57.256791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:25:57.256791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:57.256791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:57.256791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.256791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:57.257154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:57.257154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:57.257154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.257154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:57.257154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:57.257154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:57.257154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:57.257154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:57.257154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:57.257154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:57.257154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:57.257154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:57.257154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:57.257154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:57.257576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:57.257576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:57.257576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.257576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:57.257834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.257834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:57.257834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:57.257834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:57.257834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:57.257834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:57.257834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.257834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:57.257834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:57.257834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.257834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:57.257834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:57.257834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.257834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:57.257834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:57.257834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:57.258325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:57.258892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:57.258892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:57.258892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:57.258892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:57.258892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:57.258892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.258892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:57.261985 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:57.262511 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:57.262720 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:57.262866 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:57.263363 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:57.263632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.264194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:57.264194 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:57.264194 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:57.264194 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:57.264194 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:57.264194 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:57.264194 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.264194 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:57.265082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.265082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:25:57.265082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:57.265082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:57.265082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.265082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:57.265082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:57.265082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:25:57.265082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.265082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:25:57.265082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:57.265082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:25:57.265082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:57.265082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:25:57.265082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:25:57.265595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:57.265595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:25:57.265595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:57.265595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:25:57.265595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:25:57.265595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:57.265595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:25:57.265595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.265595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:25:57.266207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.266207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:57.266207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:57.266207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:57.266207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.266207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:57.266207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:57.266207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:57.266207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:57.266207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.266207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:57.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:57.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:57.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:57.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:57.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:57.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:57.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:57.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:57.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:57.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:57.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:57.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:57.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:57.266957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:57.266957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:57.266957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.266957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.267304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:57.267825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:57.268161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:57.268161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:57.268161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.268161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:25:57.270566 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:25:57.271025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:25:57.271102 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:25:57.271284 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:25:57.271797 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:57.272182 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:25:57.272313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.272937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:25:57.272937 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:57.272937 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:25:57.272937 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:25:57.272937 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:57.272937 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:25:57.272937 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.272937 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:25:57.273711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.273711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:25:57.273711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:57.273711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:25:57.273711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.273711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:57.273711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:57.273711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.273711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:57.273711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:57.273711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:25:57.273711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.274112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:25:57.274112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:57.274112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:25:57.274112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:57.274112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:25:57.274112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:25:57.274112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:57.274112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:25:57.274112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:57.274112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:25:57.274112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:25:57.274112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:57.274112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:25:57.274599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.274599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:25:57.274599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.274599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:25:57.274599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:25:57.274599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:25:57.274599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:25:57.274599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:25:57.274599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.274599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:25:57.274599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:25:57.274599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.274599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:25:57.274599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:25:57.274599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:25:57.275040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:25:57.275615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:25:57.275615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:25:57.275615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:25:57.275615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:25:57.275615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:25:57.275615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:01.793287 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:01.794130 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:01.794130 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:01.794569 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:01.831560 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:01.831808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:01.833757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:01.833757 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:01.833757 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:01.833757 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:01.833757 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:01.833757 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:01.833757 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:01.833757 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:01.833757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:01.833757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:01.833757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:01.833757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:01.833757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:01.833757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:01.833757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:01.834244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:01.834244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:01.834244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:01.834244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:01.834244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:01.834244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:01.834244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:01.834244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:01.834244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:01.834244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:01.834244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:01.834244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:01.834244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:01.834244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:01.834975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:01.834975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:01.834975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:01.834975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:01.834975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:01.834975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:01.834975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:01.834975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:01.834975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:01.834975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:01.834975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:01.834975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:01.834975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:01.834975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:01.835890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:01.836557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:01.836557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:01.836557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:01.836557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:01.836557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:01.836557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:01.836557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:01.836557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:01.836557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:02.247946 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:02.248327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:02.248527 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:02.249531 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:02.249770 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:02.249991 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:02.250152 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:02.250288 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:02.250444 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-91371b0a-0b98-46c6-b03b-2c74d3b949a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:02.250615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.251374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:02.251374 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:02.251374 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:02.251374 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:02.251374 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:02.251374 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:02.251374 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.251374 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:02.252337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.252337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:02.252337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:02.252337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:02.252337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.252337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:02.252337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:02.252337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.252337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:02.252337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:02.252337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:02.252337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.252857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:02.252857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:02.252857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:02.252857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:02.252857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:02.252857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:02.252857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:02.252857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:02.252857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:02.252857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:02.252857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:02.252857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:02.252857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:02.253227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.253227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:02.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:02.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:02.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:02.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:02.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:02.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:02.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:02.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:02.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:02.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:02.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:02.253554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:02.254030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:02.254589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:02.254589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.254589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:02.254971 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:02.255454 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:02.255613 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:02.255758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:26:02.256109 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:02.256322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.256867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:02.256867 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:02.256867 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:02.256867 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:02.256867 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:02.256867 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:02.256867 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.256867 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:02.257855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.257855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:02.257855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:02.257855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:02.257855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.257855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:02.257855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:02.257855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:02.257855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.257855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:02.257855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:02.257855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:02.257855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:02.257855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:02.257855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:02.258282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:02.258282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:02.258282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:02.258282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:02.258282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:02.258282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:02.258282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:02.258282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.258282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:02.258783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.258783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:02.258783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:02.258783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:02.258783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.258783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:02.258783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:02.258783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:26:02.258783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:02.258783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:02.258783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.258783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:02.259026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:02.259026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:02.259026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.259026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:02.259026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:02.259026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:02.259026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:02.259026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:02.259026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:02.259026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:02.259026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:02.259026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:02.259026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:02.259026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:02.259576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:02.259576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:02.259576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.259576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:02.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:02.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:02.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:02.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:02.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:02.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:02.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:02.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:02.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:02.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:02.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:02.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:02.260136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:02.260551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:02.260551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:02.260551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:02.260551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:02.260551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:02.260551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.260551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:02.262499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:02.263063 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:02.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:02.263344 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:02.263765 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:02.264683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.265284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:02.265284 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:02.265284 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:02.265284 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:02.265284 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:02.265284 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:02.265284 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.265284 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:02.266120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.266120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:02.266120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:02.266120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:02.266120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.266120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:02.266120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:02.266120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:02.266120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.266120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:02.266120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:02.266120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:02.266120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:02.266120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:02.266120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:02.266583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:02.266583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:02.266583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:02.266583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:02.266583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:02.266583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:02.266583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:02.266583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.266583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:02.267222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.267222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:02.267222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:02.267222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:02.267222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.267222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:02.267222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:02.267222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:02.267222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:02.267222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.267222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:02.267793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:02.267793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:02.267793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.267793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:02.267793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:02.267793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:02.267793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:02.267793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:02.267793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:02.267793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:02.267793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:02.267793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:02.267793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:02.267793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:02.268126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:02.268126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:02.268126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.268126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.268260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:02.268729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:02.269067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:02.269067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:02.269067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.269067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:02.271004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:02.271262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:02.271384 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:02.271608 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:02.272028 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:02.272210 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:02.272494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.273063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:02.273063 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:02.273063 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:02.273063 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:02.273063 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:02.273063 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:02.273063 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.273063 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:02.274248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.274248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:02.274248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:02.274248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:02.274248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.274248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:02.274248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:02.274248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.274248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:02.274248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:02.274248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:02.274248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.274810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:02.274810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:02.274810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:02.274810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:02.274810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:02.274810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:02.274810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:02.274810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:02.274810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:02.274810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:02.274810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:02.274810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:02.274810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:02.275082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.275082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:02.275606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.275606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:02.275606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:02.275606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:02.275606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:02.275606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:02.275606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.275606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:02.275606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:02.275606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.275606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:02.275606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:02.275606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.275606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:02.275606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:02.275606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:02.277212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:02.277212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:02.277212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:06.793707 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:06.794180 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:06.794180 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:06.794678 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:06.828999 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:06.829409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:06.830053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:06.830053 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:06.830053 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:06.830053 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:06.830053 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:06.830053 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:06.830053 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:06.830053 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:06.831111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:06.831111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:06.831111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:06.831111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:06.831111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:06.831111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:06.831111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:06.831111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:06.831111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:06.831111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:06.831111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:06.831111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:06.831361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:06.831361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:06.831361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:06.831361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:06.831361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:06.831361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:06.831361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:06.831361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:06.831361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:06.831361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:06.831361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:06.831361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:06.831361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:06.831715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:06.831715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:06.832242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:06.832242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:06.832242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:06.832242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:06.832242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:06.832242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:06.832242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:06.832242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:06.832242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:06.832242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:06.832242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:06.832242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:06.832242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:06.832242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:06.832242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:06.832242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:06.832753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:06.833150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:06.833150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:06.833150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:07.249087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:07.249087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:07.249087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:07.249787 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:07.250022 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:07.250180 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:07.250284 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:07.250424 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:07.250513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-91371b0a-0b98-46c6-b03b-2c74d3b949a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:07.250645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:07.250790 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:07.250939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:07.251090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.251719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:07.251719 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:07.251719 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:07.251719 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:07.251719 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:07.251719 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:07.251719 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.251719 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:07.252446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.252446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:07.252446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:07.252446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:07.252446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.252446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:07.252446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:07.252446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.252446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:07.252446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:07.252446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:07.252446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.252990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:07.252990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:07.252990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:07.252990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:07.252990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:07.252990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:07.252990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:07.252990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:07.252990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:07.252990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:07.252990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:07.252990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:07.252990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:07.254611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.254611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:07.254611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.254611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:07.254611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:07.254611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:07.254611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:07.254611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:07.254611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.254611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:07.254611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:07.254611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.254611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:07.254611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:07.254611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:07.255003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:07.255311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:07.255311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:07.255311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:07.255311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:07.255311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.255311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:07.255437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:07.255889 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:07.255986 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:07.256085 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:26:07.256623 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:07.256818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.257293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:07.257293 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:07.257293 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:07.257293 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:07.257293 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:07.257293 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:07.257293 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.257293 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:07.257911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.257911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:07.257911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:07.257911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:07.257911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.257911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:07.257911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:07.257911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:07.257911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.257911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:07.257911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:07.257911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:07.257911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:07.257911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:07.257911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:07.258168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:07.258168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:07.258168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:07.258168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:07.258168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:07.258168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:07.258168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:07.258168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.258168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:07.258500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.258500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:07.258500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:07.258500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:07.258500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.258500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:07.258500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:07.258500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:26:07.258500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:07.258500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:07.258500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.258500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:07.258781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:07.258781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:07.258781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.258781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:07.258781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:07.258781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:07.258781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:07.258781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:07.258781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:07.258781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:07.258781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:07.258781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:07.258781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:07.258781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:07.259037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:07.259037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:07.259037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.259037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:07.260214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.260214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:07.260214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:07.260214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:07.260214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:07.260214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:07.260214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.260214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:07.260214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:07.260214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.260214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:07.260214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:07.260214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.260214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:07.260214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:07.260214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:07.260786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:07.261281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:07.261281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:07.261281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:07.261281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:07.261281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:07.261281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.261281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:07.262966 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:07.263155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:07.263249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:07.263340 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:07.263849 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:07.264090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.264654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:07.264654 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:07.264654 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:07.264654 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:07.264654 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:07.264654 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:07.264654 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.264654 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:07.265422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.265422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:07.265422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:07.265422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:07.265422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.265422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:07.265422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:07.265422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:07.265422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.265422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:07.265422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:07.265422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:07.265422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:07.265422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:07.265422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:07.265890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:07.265890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:07.265890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:07.265890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:07.265890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:07.265890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:07.265890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:07.265890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.265890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:07.266189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.266189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:07.266189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:07.266189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:07.266189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.266189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:07.266189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:07.266189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:07.266189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:07.266189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.266189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:07.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:07.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:07.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:07.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:07.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:07.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:07.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:07.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:07.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:07.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:07.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:07.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:07.266523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:07.266931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:07.266931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:07.266931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.266931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.267052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:07.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:07.268304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:07.268304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:07.268304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.268304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:07.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:07.271951 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:07.272122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:07.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:07.272940 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:07.274061 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:07.274061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.274753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:07.274753 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:07.274753 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:07.274753 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:07.274753 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:07.274753 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:07.274753 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.274753 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:07.275929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.275929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:07.275929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:07.275929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:07.275929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.275929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:07.275929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:07.275929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.275929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:07.275929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:07.275929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:07.275929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.276330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:07.276330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:07.276330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:07.276330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:07.276330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:07.276330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:07.276330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:07.276330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:07.276330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:07.276330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:07.276330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:07.276330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:07.276330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:07.276863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.276863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:07.276863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:07.276863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:07.276863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:07.276863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:07.276863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.276863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:07.276863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:07.276863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.276863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:07.276863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:07.276863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.276863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:07.276863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:07.276863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:07.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:07.278193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:07.278193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:07.278193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:11.794134 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:11.794661 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:11.794661 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:11.795449 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:11.841076 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:11.841076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:11.841658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:11.841658 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:11.841658 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:11.841658 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:11.841658 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:11.841658 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:11.841658 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:11.841658 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:11.842587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:11.842587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:11.842587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:11.842587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:11.842587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:11.842587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:11.842587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:11.842587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:11.842587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:11.842587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:11.842587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:11.842587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:11.842899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:11.842899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:11.842899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:11.842899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:11.842899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:11.842899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:11.842899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:11.842899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:11.842899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:11.842899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:11.842899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:11.842899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:11.842899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:11.843169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:11.843169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:11.843839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:11.843839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:11.843839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:11.843839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:11.843839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:11.843839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:11.843839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:11.843839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:11.843839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:11.843839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:11.843839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:11.843839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:11.843839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:11.843839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:11.843839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:11.843839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:11.844322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:11.844841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:11.844841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:11.844841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:12.249633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:12.249952 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:12.250189 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:12.251774 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:12.251986 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:12.252199 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:12.252372 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:12.252553 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:12.252717 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:12.252873 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:12.253025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-91371b0a-0b98-46c6-b03b-2c74d3b949a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:12.253177 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:12.253329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.253910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:12.253910 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:12.253910 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:12.253910 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:12.253910 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:12.253910 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:12.253910 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.253910 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:12.255155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.255155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:12.255155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:12.255155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:12.255155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.255155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:12.255155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:12.255155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.255155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:12.255155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:12.255155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:12.255155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.255435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:12.255435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:12.255435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:12.255435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:12.255435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:12.255435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:12.255435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:12.255435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:12.255435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:12.255435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:12.255435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:12.255435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:12.255435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:12.255767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.255767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:12.260295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.260295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:12.260295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:12.260295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:12.260295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:12.260295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:12.260295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.260295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:12.260295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:12.260295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.260295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:12.260295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:12.260295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.260295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:12.260295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:12.260295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:12.261044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:12.261642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:12.261642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.261642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:12.261642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:12.261642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:12.261642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:12.261642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:26:12.261642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:12.261889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.263468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:12.264066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:12.264066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:12.264066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:12.264066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:12.264066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:12.264066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:12.264066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:12.264066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:12.264066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:12.264066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:12.264066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:12.264066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:12.264066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.264066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:12.266115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.266115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:12.266115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:12.266115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:12.266115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.266115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:12.266115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:12.266115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:26:12.266115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:12.266115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:12.266115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.266115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:12.266417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:12.266417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:12.266417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.266417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:12.266417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:12.266417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:12.266417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:12.266417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:12.266417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:12.266417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:12.266417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:12.266417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:12.266417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:12.266417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:12.266668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:12.266668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:12.266668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.266668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:12.266874 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:12.267718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.267718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:12.267718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:12.267718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:12.267718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:12.267718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:12.267718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.267718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:12.267718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:12.267718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.267718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:12.267718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:12.267718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.267718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:12.267718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:12.267718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:12.268057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:12.268399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:12.268399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:12.268399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:12.268399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:12.268399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:12.268399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.268399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:12.268399 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:12.268399 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:12.268399 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:12.269939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:12.270207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.270759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:12.270759 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:12.270759 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:12.270759 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:12.270759 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:12.270759 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:12.270759 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.270759 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:12.271739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.271739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:12.271739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:12.271739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:12.271739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.271739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:12.271739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:12.271739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:12.271739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.271739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:12.271739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:12.271739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:12.271739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:12.271739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:12.271739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:12.272197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:12.272197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:12.272197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:12.272197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:12.272197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:12.272197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:12.272197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:12.272197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.272197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:12.272197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.272197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:12.272197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:12.272197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:12.272197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.272791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:12.272791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:12.272791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:12.272791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:12.272791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.272791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:12.272791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:12.272791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:12.272791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.272791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:12.272791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:12.272791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:12.272791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:12.273210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:12.273210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:12.273210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:12.273210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:12.273210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:12.273210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:12.273210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:12.273210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:12.273210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:12.273210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.273210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:12.273210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:12.273500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:12.274657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:12.274657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:12.274657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:12.274657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:12.274657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:12.274657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:12.276346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.276346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:12.276346 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:12.276346 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:12.276346 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:12.276346 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:12.276346 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:12.276346 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.276346 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:12.276346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.276346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:12.276346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:12.276346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:12.276346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.276346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:12.276845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:12.276845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.276845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:12.276845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:12.276845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:12.276845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.276845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:12.276845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:12.276845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:12.276845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:12.276845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:12.276845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:12.276845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:12.276845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:12.277283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:12.277283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:12.277283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:12.277283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:12.277283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:12.277283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.277283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:12.277283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.277283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:12.277283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:12.277283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:12.277283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:12.277283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:12.277857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:12.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:12.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:12.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:12.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:12.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:12.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:12.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:12.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:12.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:12.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:12.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:12.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:16.794736 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:16.795499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:16.795778 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:16.796858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:16.844398 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:16.844398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:16.844850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:16.844850 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:16.844850 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:16.844850 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:16.844850 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:16.844850 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:16.844850 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:16.844850 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:16.846098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.049 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:16.846098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:16.846098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:16.846098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:16.846098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:16.846098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:16.846098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:16.846098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:16.846098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:16.846098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:16.846098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:16.846098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:16.846456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:16.846456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:16.846456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:16.846456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:16.846456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:16.846456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:16.846456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:16.846456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:16.846456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:16.846456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:16.846456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:16.846456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:16.846456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:16.846713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:16.846713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:16.848586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:16.848586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:16.848586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:16.848586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:16.848586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:16.848586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:16.848586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:16.848586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:16.848586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:16.848586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:16.848586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:16.848586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:16.848586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:16.848586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:16.848586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:16.848586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:16.848929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:16.849213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:16.849213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:16.849213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:17.249527 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:17.250108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:17.250241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:17.251328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:17.251521 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:17.251685 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:17.251817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:17.251942 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-91371b0a-0b98-46c6-b03b-2c74d3b949a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:17.252077 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:17.252203 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:17.252331 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:17.252493 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:17.252643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.253189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:17.253189 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:17.253189 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:17.253189 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:17.253189 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:17.253189 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:17.253189 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.253189 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:17.254157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.254157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:17.254157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:17.254157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:17.254157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.254157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:17.254157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:17.254157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.254157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:17.254157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:17.254157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:17.254157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.254451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:17.254451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:17.254451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:17.254451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:17.254451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:17.254451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:17.254451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:17.254451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:17.254451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:17.254451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:17.254451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:17.254451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:17.254451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:17.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.254683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:17.255103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.255103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:17.255103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:17.255103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:17.255103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:17.255103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:17.255103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.255103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:17.255103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:17.255103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.255103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:17.255103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:17.255103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.255103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:17.255103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:17.255103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:17.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:17.255786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:17.255786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.255786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:17.260862 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:17.261320 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:17.261492 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:17.261617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:26:17.261985 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:17.262185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.262789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:17.262789 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:17.262789 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:17.262789 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:17.262789 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:17.262789 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:17.262789 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.262789 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:17.263773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.263773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:17.263773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:17.263773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:17.263773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.263773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:17.263773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:17.263773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:17.263773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.263773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:17.263773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:17.263773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:17.263773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:17.263773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:17.263773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:17.264333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:17.264333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:17.264333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:17.264333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:17.264333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:17.264333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:17.264333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:17.264333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.264333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:26:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:17.265059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:17.265059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:17.265059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.265059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:17.265059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:17.265059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:17.265059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:17.265059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:17.265059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:17.265059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:17.265059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:17.265059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:17.265059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:17.265059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:17.265469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:17.265469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:17.265469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.265469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:17.266008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.266008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:17.266008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:17.266008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:17.266008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:17.266008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:17.266008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.266008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:17.266008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:17.266008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.266008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:17.266008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:17.266008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.266008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:17.266008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:17.266008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:17.266825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:17.267323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:17.267323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:17.267323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:17.267323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:17.267323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:17.267323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.267323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:17.268112 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:17.268532 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:17.268823 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:17.268941 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:17.269306 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:17.269617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.270282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:17.270282 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:17.270282 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:17.270282 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:17.270282 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:17.270282 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:17.270282 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.270282 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:17.271069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.271069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:17.271069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:17.271069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:17.271069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.271069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:17.271069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:17.271069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:17.271069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.271069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:17.271069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:17.271069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:17.271069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:17.271069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:17.271069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:17.271601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:17.271601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:17.271601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:17.271601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:17.271601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:17.271601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:17.271601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:17.271601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.271601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:17.272066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.272066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:17.272066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:17.272066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:17.272066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.272066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:17.272066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:17.272066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:17.272066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:17.272066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.272066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:17.272448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:17.272448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:17.272448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.272448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:17.272448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:17.272448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:17.272448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:17.272448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:17.272448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:17.272448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:17.272448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:17.272448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:17.272448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:17.272448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:17.272872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:17.272872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:17.272872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.272872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.273295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:17.273997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:17.274617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:17.274617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:17.274617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.274617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:17.274617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:17.274617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:17.274617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:17.274617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:17.274871 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:17.274871 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:17.275084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:17.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:17.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:17.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:17.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:17.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:17.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:17.277696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.277696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:17.277696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:17.277696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:17.277696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.277696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:17.277696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:17.277696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.277696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:17.277696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:17.277696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:17.277696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.278142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:17.278142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:17.278142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:17.278142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:17.278142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:17.278142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:17.278142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:17.278142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:17.278142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:17.278142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:17.278142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:17.278142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:17.278142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:17.278661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.278661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:17.278760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.278760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:17.278760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:17.278760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:17.278760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:17.278760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:17.278760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.278760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:17.278760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:17.278760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.278760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:17.278760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:17.278760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.278760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:17.278760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:17.278760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:17.279211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:17.279867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:17.279867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:17.279867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:21.795984 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:21.796502 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:21.796502 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:21.797333 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:21.798305 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=240 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:26:21.835894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:21.836175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:21.836841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:21.836841 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:21.836841 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:21.836841 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:21.836841 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:21.836841 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:21.836841 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:21.836841 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:21.837591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:21.837591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:21.837591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:21.837591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:21.837591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:21.837591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:21.837591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:21.837591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:21.837591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:21.837591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:21.837591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:21.837591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:21.837880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:21.837880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:21.837880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:21.837880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:21.837880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:21.837880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:21.837880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:21.837880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:21.837880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:21.837880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:21.837880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:21.837880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:21.837880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:21.838091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:21.838091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:21.838470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:21.838470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:21.838470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:21.838470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:21.838470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:21.838470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:21.838470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:21.838470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:21.838470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:21.838470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:21.838470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:21.838470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:21.838470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:21.838470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:21.838470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:21.838470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:21.839056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:21.839611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:21.839611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:21.839611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:22.250191 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:22.250547 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:22.250754 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:22.252638 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:22.252870 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:22.253039 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:22.253195 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:22.253357 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-91371b0a-0b98-46c6-b03b-2c74d3b949a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:22.253530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:22.255853 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:22.256014 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:22.256175 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:22.256338 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:22.256468 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:22.257948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-714aff1c-85de-462e-a525-1bfe2ba61b53, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:22.258119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.258703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:22.258703 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:22.258703 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:22.258703 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:22.258703 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:22.258703 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:22.258703 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.258703 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:22.259483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.259483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:22.259483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:22.259483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:22.259483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.259483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:22.259483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:22.259483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.259483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:22.259483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:22.259483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:22.259483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.259739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:22.259739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:22.259739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:22.259739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:22.259739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:22.259739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:22.259739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:22.259739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:22.259739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:22.259739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:22.259739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:22.259739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:22.259739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:22.260038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.260038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:22.260444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.260444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:22.260444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:22.260444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:22.260444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:22.260444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:22.260444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.260444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:22.260444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:22.260444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.260444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:22.260444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:22.260444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.260444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:22.260444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:22.260444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:22.261527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:22.261964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:22.261964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.261964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:22.261964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:22.261964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:22.261964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:22.261964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:26:22.262144 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:22.262507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.263589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:22.263589 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:22.263589 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:22.263589 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:22.263589 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:22.263589 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:22.263589 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.263589 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:22.266358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.266358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:22.266358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:22.266358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:22.266358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.266358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:22.266358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:22.266358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:22.266358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.266358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:22.266358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:22.266358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:22.266358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:22.266358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:22.266358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:22.267077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:22.267077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:22.267077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:22.267077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:22.267077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:22.267077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:22.267077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:22.267077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.267077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:22.268850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.268850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:22.268850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:22.268850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:22.268850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.268850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:22.268850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:22.268850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:26:22.268850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:22.268850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:22.268850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.268850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:22.269229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:22.269229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:22.269229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.269229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:22.269229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:22.269229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:22.269229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:22.269229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:22.269229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:22.269229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:22.269229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:22.269229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:22.269229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:22.269229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:22.270046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:22.270046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:22.270046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.270046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:22.272185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.272185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:22.272185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:22.272185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:22.272185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:22.272185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:22.272185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.272185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:22.272185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:22.272185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.272185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:22.272185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:22.272185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.272185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:22.272185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:22.272185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:22.273782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:22.273782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:22.273782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:22.273782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:22.273782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:22.273782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.273782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:22.273782 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:22.273782 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:22.273782 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:22.273970 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:22.273970 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:22.273970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.274821 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.276907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:22.277560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:22.277560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:22.277560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:22.277560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:22.277560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:22.277560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:22.277560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:22.277560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:22.277560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:22.277560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:22.277560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:22.277560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:22.277560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.277560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:22.277560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.277953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:22.277953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:22.277953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:22.277953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.277953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:22.277953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:22.277953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:22.277953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:22.277953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.277953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:22.277953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:22.277953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:22.277953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.278307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:22.278307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:22.278307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:22.278307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:22.278307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:22.278307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:22.278307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:22.278307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:22.278307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:22.278307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:22.278307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:22.278307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:22.278307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:22.278660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.278660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:22.278660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.278660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:22.278660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:22.278660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:22.278660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:22.278660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:22.278660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.278660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:22.278660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:22.278660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.278660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:22.278660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:22.278660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:22.278916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:22.279210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:22.279210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:22.279210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:22.279210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:22.279210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:22.279210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:22.279210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.279210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:22.279210 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:22.279210 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:22.279210 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:22.279765 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:22.279765 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:22.279765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.279765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:22.279765 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:22.279765 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:22.279765 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:22.279765 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:22.279765 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:22.279765 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.279765 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:22.280088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.280088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:22.280088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:22.280088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:22.280088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.280088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:22.280088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:22.280088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.280088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:22.280088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:22.280088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:22.280088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.280292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:22.280292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:22.280292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:22.280292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:22.280292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:22.280292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:22.280292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:22.280292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:22.280292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:22.280292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:22.280292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:22.280292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:22.280292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:22.280599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.280599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:22.280868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.280868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:22.280868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:22.280868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:22.280868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:22.280868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:22.280868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.280868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:22.280868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:22.280868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.280868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:22.280868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:22.280868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.280868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:22.280868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:22.280868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:22.282100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:22.282440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:22.282440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:22.282440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:26.796758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:26.797194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:26.797234 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:26.797992 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:26.840943 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:26.841151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:26.843322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:26.843322 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:26.843322 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:26.843322 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:26.843322 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:26.843322 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:26.843322 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:26.843322 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:26.843322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:26.843322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:26.843322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:26.843322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:26.843322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:26.843322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:26.843322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:26.843950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:26.843950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:26.843950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:26.843950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:26.843950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:26.843950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:26.843950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:26.843950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:26.843950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:26.843950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:26.843950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:26.843950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:26.843950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:26.843950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:26.844347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:26.844347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:26.844347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:26.844347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:26.844347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:26.844347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:26.844347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:26.844347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:26.844347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:26.844347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:26.844347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:26.844347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:26.844347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:26.844347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:26.844664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:26.845034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:26.845034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:26.845034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:26.845034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:26.845034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:26.845034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:26.845034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:26.845034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:26.845034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:27.250666 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:27.251452 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:27.251652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:27.254055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.254259 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.254405 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.254548 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.254706 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-91371b0a-0b98-46c6-b03b-2c74d3b949a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.254869 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.254992 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.255108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.255244 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.255370 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.255532 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-714aff1c-85de-462e-a525-1bfe2ba61b53, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.255695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.255804 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.255947 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.256069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.257495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:27.257495 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:27.257495 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:27.257495 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:27.257495 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:27.257495 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:27.257495 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.257495 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:27.258698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.258698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:27.258698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:27.258698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:27.258698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.258698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:27.258698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:27.258698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.258698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:27.258698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:27.258698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:27.258698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.259086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:27.259086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:27.259086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:27.259086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:27.259086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:27.259086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:27.259086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:27.259086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:27.259086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:27.259086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:27.259086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:27.259086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:27.259086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:27.259501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.259501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:27.259831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.259831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:27.259831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:27.259831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:27.259831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:27.259831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:27.259831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.259831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:27.259831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:27.259831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.259831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:27.259831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:27.259831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.259831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:27.259831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:27.259831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:27.260330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:27.260965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:27.260965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.260965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:27.261720 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:27.263935 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:27.264131 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:27.264303 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:26:27.264831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.265045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.267109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:27.267109 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:27.267109 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:27.267109 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:27.267109 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:27.267109 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:27.267109 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.267109 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:27.267809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.267809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:27.267809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:27.267809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:27.267809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.267809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:27.267809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:27.267809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:27.267809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.267809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:27.267809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:27.267809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:27.267809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:27.267809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:27.267809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:27.268105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:27.268105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:27.268105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:27.268105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:27.268105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:27.268105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:27.268105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:27.268105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.268105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:27.268627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.268627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:27.268627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:27.268627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:27.268627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.268627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:27.268627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:27.268627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:26:27.268627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:27.268627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:27.268627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.268627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:27.268855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:27.268855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:27.268855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.268855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:27.268855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:27.268855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:27.268855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:27.268855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:27.268855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:27.268855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:27.268855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:27.268855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:27.268855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:27.268855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:27.269097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:27.269097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:27.269097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.269097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:27.269554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.269554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:27.269554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:27.269554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:27.269554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:27.269554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:27.269554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.269554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:27.269554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:27.269554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.269554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:27.269554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:27.269554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.269554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:27.269554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:27.269554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:27.270025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:27.270359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:27.270359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:27.270359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:27.270359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:27.270359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:27.270359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.270359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:27.272951 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:27.273163 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:27.273257 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:27.273367 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:27.273834 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.274068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.274557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:27.274557 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:27.274557 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:27.274557 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:27.274557 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:27.274557 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:27.274557 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.274557 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:27.275111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.275111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:27.275111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:27.275111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:27.275111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.275111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:27.275111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:27.275111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:27.275111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.275111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:27.275111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:27.275111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:27.275111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:27.275111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:27.275111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:27.275419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:27.275419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:27.275419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:27.275419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:27.275419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:27.275419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:27.275419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:27.275419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.275419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:27.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:27.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:27.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:27.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:27.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:27.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:27.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:27.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.275883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:27.276168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:27.276168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:27.276168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.276168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:27.276168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:27.276168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:27.276168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:27.276168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:27.276168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:27.276168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:27.276168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:27.276168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:27.276168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:27.276168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:27.276477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:27.276477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:27.276477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.276477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.276769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:27.277214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:27.277638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:27.277638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:27.277638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.277638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:27.277841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:27.278007 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:27.278071 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:27.278198 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:27.278637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.278847 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:27.279143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.280003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:27.280003 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:27.280003 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:27.280003 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:27.280003 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:27.280003 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:27.280003 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.280003 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:27.280713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.280713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:27.280713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:27.280713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:27.280713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.280713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:27.280713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:27.280713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.280713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:27.280713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:27.280713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:27.280713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.280958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:27.280958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:27.280958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:27.280958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:27.280958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:27.280958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:27.280958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:27.280958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:27.280958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:27.280958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:27.280958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:27.280958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:27.280958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:27.281222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.281222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:27.281482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.281482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:27.281482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:27.281482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:27.281482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:27.281482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:27.281482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.281482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:27.281482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:27.281482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.281482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:27.281482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:27.281482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.281482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:27.281482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:27.281482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:27.281927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:27.282304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:27.282304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:27.282304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:31.797190 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:31.799456 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:31.799877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:31.801930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:31.856837 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:31.857116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:31.859705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:31.859705 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:31.859705 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:31.859705 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:31.859705 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:31.859705 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:31.859705 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:31.859705 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:31.859705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:31.859705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:31.859705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:31.859705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:31.859705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:31.859705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:31.859705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:31.860224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:31.860224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:31.860224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:31.860224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:31.860224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:31.860224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:31.860224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:31.860224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:31.860224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:31.860224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:31.860224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:31.860224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:31.860224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:31.860224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:31.860557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:31.860557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:31.860557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:31.860557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:31.860557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:31.860557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:31.860557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:31.860557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:31.860557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:31.860557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:31.860557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:31.860557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:31.860557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:31.860557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:31.860840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:31.861318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:31.861318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:31.861318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:31.861318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:31.861318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:31.861318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:31.861318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:31.861318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:31.861318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:32.251444 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:32.251808 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:32.251935 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:32.253087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.253273 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.253449 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.253643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.253814 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-91371b0a-0b98-46c6-b03b-2c74d3b949a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.254271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.254271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.254271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.254271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.254492 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.255559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.255559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-714aff1c-85de-462e-a525-1bfe2ba61b53, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.255559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.255559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.255559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.255559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:32.255559 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:32.255559 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:32.255559 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:32.255841 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:32.255841 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:32.255841 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.255841 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:32.256529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.256529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:32.256529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:32.256529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:32.256529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.256529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:32.256529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:32.256529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.256529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:32.256529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:32.256529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:32.256529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.256838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:32.256838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:32.256838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:32.256838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:32.256838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:32.256838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:32.256838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:32.256838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:32.256838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:32.256838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:32.256838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:32.256838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:32.256838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:32.257057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.257057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:32.257636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.257636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:32.257636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:32.257636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:32.257636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:32.257636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:32.257636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.257636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:32.257636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:32.257636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.257636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:32.257636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:32.257636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.257636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:32.257636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:32.257636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:32.257981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:32.258766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:32.258766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.258766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:32.262519 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:32.262933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:32.263113 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:32.263277 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:26:32.264182 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.264182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.264308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:32.264308 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:32.264308 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:32.264308 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:32.264308 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:32.264308 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:32.264308 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.264308 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:32.265835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.265835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:32.265835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:32.265835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:32.265835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.265835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:32.265835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:32.265835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:32.265835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.265835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:32.265835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:32.265835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:32.265835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:32.265835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:32.265835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:32.266371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:32.266371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:32.266371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:32.266371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:32.266371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:32.266371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:32.266371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:32.266371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.266371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:32.266371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.266371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:32.266371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:32.266371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:32.266371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.266868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:32.266868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:32.266868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:26:32.266868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:32.266868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:32.266868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.266868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:32.266868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:32.266868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:32.266868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.266868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:32.266868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:32.266868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:32.267130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:32.267130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:32.267130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:32.267130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:32.267130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:32.267130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:32.267130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:32.267130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:32.267130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:32.267130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:32.267130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.267130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:32.267411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.267411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:32.267411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:32.267411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:32.267411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:32.267411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:32.267411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.267411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:32.267411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:32.267411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.267411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:32.267411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:32.267411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.267411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:32.267411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:32.267411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:32.267752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:32.268164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:32.268164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:32.268164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:32.268164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:32.268164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:32.268164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.268164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:32.273283 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:32.273662 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:32.273780 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:32.273921 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:32.274469 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:32.276629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:32.277189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.277189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:32.277189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:32.277189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:32.277189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:32.277189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:32.277189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:32.277189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:32.277189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:32.277189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:32.277189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:32.277189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:32.277189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:32.277189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:32.277189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.277189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:32.277733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.277733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:32.277733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:32.277733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:32.277733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.277733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:32.277733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:32.277733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:32.277733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:32.277733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.277733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:32.278030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:32.278030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:32.278030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.278030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:32.278030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:32.278030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:32.278030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:32.278030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:32.278030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:32.278030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:32.278030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:32.278030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:32.278030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:32.278030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:32.278459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:32.278459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:32.278459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.278459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:32.278459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.278459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:32.278459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:32.278459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:32.278459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:32.278459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:32.278459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.278459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:32.278459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:32.278459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:32.278917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:32.281652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:32.281652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:32.281652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:32.281652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:32.281652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:32.281652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:32.281652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:32.281652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:32.281652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:32.281652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.281652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:32.281652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:32.281652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:32.282209 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:32.282209 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:32.282209 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.282209 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:32.282209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.282209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:32.282209 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:32.282209 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:32.282209 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:32.282519 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:32.282519 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:32.282519 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.282519 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:32.282704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.282704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:32.282704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:32.282704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:32.282704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.282704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:32.282704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:32.282704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.282704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:32.282704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:32.282704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:32.282704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.283135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:32.283135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:32.283135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:32.283135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:32.283135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:32.283135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:32.283135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:32.283135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:32.283135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:32.283135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:32.283135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:32.283135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:32.283135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:32.283525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.283525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:32.283844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.283844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:32.283844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:32.283844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:32.283844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:32.283844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:32.283844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.283844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:32.283844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:32.283844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.283844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:32.283844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:32.283844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.283844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:32.283844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:32.283844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:32.284428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:32.284998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:32.284998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:32.284998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:36.800003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:36.800003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:36.801582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:36.801582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:36.863043 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:36.863265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:36.863749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:36.863749 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:36.863749 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:36.863749 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:36.863749 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:36.863749 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:36.863749 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:36.863749 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:36.864575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.064 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:36.864575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:36.864575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:36.864575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:36.864575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:36.864575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:36.864575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:36.864575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:36.864575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:36.864575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:36.864575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:36.864575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:36.865802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:36.865802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:36.865802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:36.865802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:36.865802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:36.865802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:36.865802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:36.865802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:36.865802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:36.865802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:36.865802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:36.865802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:36.865802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:36.866251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:36.866251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:36.866251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:36.866251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:36.866251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:36.866251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:36.866251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:36.866251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:36.866251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:36.866251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:36.866251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:36.866251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:36.866251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:36.866251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:36.866251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:36.866641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:36.867089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:36.867089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:36.867089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:36.867089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:36.867089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:36.867089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:37.251975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:37.253314 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:37.253314 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:37.256175 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.256449 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.256902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.256902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.257006 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-91371b0a-0b98-46c6-b03b-2c74d3b949a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.257163 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.257327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.257513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.257706 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.257881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.258050 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-714aff1c-85de-462e-a525-1bfe2ba61b53, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.258423 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.258423 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.259653 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.260074 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-01b0936e-ffbb-4414-80a1-4711a57d63a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.261393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.263094 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:37.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:37.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:37.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:37.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:37.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:37.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:37.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.263621 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:37.267078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.011 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.267078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:37.267078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:37.267078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:37.267078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.267078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:37.267078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:37.267078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.267078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:37.267078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:37.267078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:37.267078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.267751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:37.267751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:37.267751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:37.267751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:37.267751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:37.267751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:37.267751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:37.267751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:37.267751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:37.267751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:37.267751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:37.267751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:37.267751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:37.268061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.268061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:37.268061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.268061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:37.268061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:37.268061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:37.268061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:37.268061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:37.268061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.268061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:37.268061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:37.268061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.268061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:37.268061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:37.268061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:37.268454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:37.268902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:37.268902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:37.268902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:37.268902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:37.268902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.268902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:37.268902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:37.268902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:37.268902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:26:37.269354 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.269354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.269830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:37.269830 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:37.269830 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:37.269830 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:37.269830 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:37.269830 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:37.269830 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.269830 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:37.271879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.271879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:37.271879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:37.271879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:37.271879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.271879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:37.271879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:37.271879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:37.271879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.271879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:37.271879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:37.271879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:37.271879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:37.271879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:37.271879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:37.272343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:37.272343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:37.272343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:37.272343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:37.272343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:37.272343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:37.272343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:37.272343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.272343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:37.274506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.274506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:37.274506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:37.274506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:37.274506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.274506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:37.274506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:37.274506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:26:37.274506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:37.274506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:37.274506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.274506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:37.274986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:37.274986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:37.274986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.274986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:37.274986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:37.274986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:37.274986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:37.274986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:37.274986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:37.274986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:37.274986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:37.274986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:37.274986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:37.274986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:37.275229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:37.275229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:37.275229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.275229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:37.275229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.275229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:37.275229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:37.275229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:37.275229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:37.275229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:37.275229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.275229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:37.275229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:37.275229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:37.275632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:37.276130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:37.276130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:37.276130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:37.276130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:37.276130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:37.276130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:37.276130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:37.276130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:37.276130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:37.276130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:37.276130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.276130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:37.276130 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:37.276130 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:37.276598 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:37.276598 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:37.276598 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.276598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.276794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:37.276794 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:37.276794 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:37.276794 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:37.276794 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:37.276794 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:37.276794 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.276794 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:37.278260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.278260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:37.278260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:37.278260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:37.278260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.278260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:37.278260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:37.278260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:37.278260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.278260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:37.278260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:37.278260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:37.278260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:37.278260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:37.278260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:37.278686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:37.278686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:37.278686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:37.278686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:37.278686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:37.278686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:37.278686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:37.278686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.278686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:37.278686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.278686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:37.278686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:37.278686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:37.278686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.279060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:37.279060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:37.279060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:37.279060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:37.279060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.279060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:37.279060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:37.279060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:37.279060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.279060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:37.279060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:37.279060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:37.279060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:37.279338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:37.279338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:37.279338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:37.279338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:37.279338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:37.279338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:37.279338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:37.279338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:37.279338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:37.279338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.279338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:37.279338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:37.279646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.280148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:37.280752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:37.280752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:37.280752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:37.280752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:37.280849 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.281090 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:37.281446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.281969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:37.281969 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:37.281969 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:37.281969 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:37.281969 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:37.281969 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:37.281969 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.281969 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:37.282779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.282779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:37.282779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:37.282779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:37.282779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.282779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:37.282779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:37.282779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.282779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:37.282779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:37.282779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:37.282779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.283258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:37.283258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:37.283258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:37.283258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:37.283258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:37.283258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:37.283258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:37.283258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:37.283258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:37.283258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:37.283258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:37.283258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:37.283258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:37.283747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.283747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:37.284058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.284058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:37.284058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:37.284058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:37.284058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:37.284058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:37.284058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.284058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:37.284058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:37.284058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.284058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:37.284058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:37.284058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.284058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:37.284058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:37.284058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:37.284514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:37.284901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:37.284901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:37.284901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:41.800097 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:41.800618 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:41.800760 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:41.801497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:41.857016 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:41.857016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:41.857491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:41.857491 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:41.857491 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:41.857491 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:41.857491 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:41.857491 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:41.857491 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:41.857491 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:41.858557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:41.858557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:41.858557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:41.858557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:41.858557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:41.858557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:41.858557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:41.858557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:41.858557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:41.858557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:41.858557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:41.858557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:41.858557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:41.858784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:41.858784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:41.859433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:41.859433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:41.859433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:41.859433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:41.859433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:41.859433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:41.859433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:41.859433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:41.859433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:41.859433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:41.859433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:41.859433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:41.859433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:41.859433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:41.859433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:41.859433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:41.859782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:41.860154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:41.860154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:41.860154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:42.252978 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:42.254012 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:42.254012 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:42.255802 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.256113 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.256275 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.256377 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.256517 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-91371b0a-0b98-46c6-b03b-2c74d3b949a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.256634 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.256759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.256855 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.256950 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.257045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.257142 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-714aff1c-85de-462e-a525-1bfe2ba61b53, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.257235 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.257330 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.257465 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.257615 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-01b0936e-ffbb-4414-80a1-4711a57d63a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.257800 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=15): DelStaticRoutesCommand(_result=None, lrouter=neutron-49504799-a156-4a0d-8b77-bb5f003180c1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.257887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.258492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:42.258492 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:42.258492 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:42.258492 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:42.258492 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:42.258492 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:42.258492 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.258492 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:42.259254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.259254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:42.259254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:42.259254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:42.259254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.259254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:42.259254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:42.259254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.259254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:42.259254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:42.259254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:42.259254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.259682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:42.259682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:42.259682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:42.259682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:42.259682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:42.259682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:42.259682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:42.259682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:42.259682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:42.259682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:42.259682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:42.259682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:42.259682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:42.260017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.260017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:42.260119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.260119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:42.260119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:42.260119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:42.260119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:42.260119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:42.260119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.260119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:42.260119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:42.260119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.260119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:42.260119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:42.260119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.260119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:42.260119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:42.260119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:42.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:42.261180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:42.261180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.261180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:42.267870 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:42.268223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:42.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:42.268505 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:26:42.268951 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.269137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.269647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:42.269647 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:42.269647 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:42.269647 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:42.269647 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:42.269647 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:42.269647 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.269647 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:42.270422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.270422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:42.270422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:42.270422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:42.270422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.270422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:42.270422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:42.270422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:42.270422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.270422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:42.270422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:42.270422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:42.270422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:42.270422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:42.270422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:42.270927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:42.270927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:42.270927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:42.270927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:42.270927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:42.270927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:42.270927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:42.270927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.270927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:42.271189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.271189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:42.271189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:42.271189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:42.271189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.271189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:42.271189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:42.271189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:26:42.271189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:42.271189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:42.271189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.271189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:42.271618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:42.271618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:42.271618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.271618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:42.271618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:42.271618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:42.271618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:42.271618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:42.271618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:42.271618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:42.271618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:42.271618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:42.271618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:42.271618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:42.272228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:42.272228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:42.272228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.272228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:42.272228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.272228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:42.272228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:42.272228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:42.272228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:42.272228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:42.272228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.272228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:42.272228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:42.272228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:42.272655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:42.273244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:42.273244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:42.273244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:42.273244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:42.273244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:42.273244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:42.273244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:42.273244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:42.273244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:42.273244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:42.273244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.273244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:42.275806 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:42.276025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:42.276120 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:42.276229 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:42.276795 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.277135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.277719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:42.277719 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:42.277719 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:42.277719 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:42.277719 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:42.277719 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:42.277719 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.277719 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:42.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:42.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:42.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:42.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:42.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:42.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:42.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:42.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:42.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:42.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:42.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:42.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:42.278819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:42.278819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:42.278819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:42.278819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:42.278819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:42.278819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:42.278819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:42.278819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.278819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:42.279160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.279160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:42.279160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:42.279160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:42.279160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.279160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:42.279160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:42.279160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:42.279160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:42.279160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.279160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:42.279496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:42.279496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:42.279496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.279496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:42.279496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:42.279496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:42.279496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:42.279496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:42.279496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:42.279496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:42.279496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:42.279496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:42.279496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:42.279496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:42.280053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:42.280053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:42.280053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.280053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:42.280053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.280053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:42.280053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:42.280053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:42.280053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:42.280053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:42.280053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.280053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:42.280053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:42.280053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:42.280449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:42.281004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:42.281004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:42.281004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:42.281004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:42.281004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:42.281004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:42.281004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:42.281004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:42.281004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:42.281004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.281004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:42.281004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:42.281004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:42.281369 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:42.281369 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:42.281369 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.281369 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:42.281524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.281818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:42.281818 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:42.281818 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:42.281818 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:42.281818 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:42.281818 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:42.281818 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.281818 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:42.282455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.282455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:42.282455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:42.282455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:42.282455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.282455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:42.282455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:42.282455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.282455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:42.282455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:42.282455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:42.282455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.282868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:42.282868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:42.282868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:42.282868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:42.282868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:42.282868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:42.282868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:42.282868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:42.282868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:42.282868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:42.282868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:42.282868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:42.282868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:42.283159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.283159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:42.283159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.283159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:42.283159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:42.283159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:42.283159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:42.283159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:42.283159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.283159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:42.283159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:42.283159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.283159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:42.283159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:42.283159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:42.283569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:42.284048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:42.284048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:42.284048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:42.284048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:42.284048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:42.284048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:46.800533 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:46.800989 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:46.800989 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:46.801806 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:46.845072 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:46.845343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:46.846039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:46.846039 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:46.846039 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:46.846039 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:46.846039 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:46.846039 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:46.846039 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:46.846039 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:46.847094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:46.847094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:46.847094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:46.847094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:46.847094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:46.847094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:46.847094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:46.847094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:46.847094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:46.847094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:46.847094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:46.847094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:46.847417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:46.847417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:46.847417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:46.847417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:46.847417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:46.847417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:46.847417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:46.847417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:46.847417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:46.847417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:46.847417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:46.847417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:46.847417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:46.847649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:46.847649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:46.848230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:46.848230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:46.848230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:46.848230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:46.848230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:46.848230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:46.848230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:46.848230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:46.848230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:46.848230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:46.848230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:46.848230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:46.848230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:46.848230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:46.848230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:46.848230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:46.849129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:46.849675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:46.849675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:46.849675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:47.253579 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:47.254156 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:47.254156 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:47.255578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.255839 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.256027 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.256188 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.256357 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.256524 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.256693 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.256840 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.256985 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.257129 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-714aff1c-85de-462e-a525-1bfe2ba61b53, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.257646 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.257646 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.257646 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.258073 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-01b0936e-ffbb-4414-80a1-4711a57d63a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.258073 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-49504799-a156-4a0d-8b77-bb5f003180c1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.258073 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=15): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.258252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=16): DelStaticRoutesCommand(_result=None, lrouter=neutron-91371b0a-0b98-46c6-b03b-2c74d3b949a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.258305 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=17): DelStaticRoutesCommand(_result=None, lrouter=neutron-6dd139ce-2ca3-470e-b5db-f6d77d05e04c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.260337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.260337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:47.260337 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:47.260337 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:47.260337 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:47.260337 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:47.260337 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:47.260337 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.260337 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:47.260337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.260337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:47.260337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:47.260337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:47.260337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.260337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:47.260716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:47.260716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.260716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:47.260716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:47.260716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:47.260716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.260716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:47.260716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:47.260716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:47.260716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:47.260716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:47.260716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:47.260716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:47.260716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:47.260929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:47.260929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:47.260929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:47.260929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:47.260929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:47.260929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.260929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:47.261040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.261040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:47.261040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:47.261040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:47.261040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:47.261040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:47.261040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.261040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:47.261040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:47.261040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.261040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:47.261040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:47.261040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.261040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:47.261040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:47.261040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:47.261283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:47.261660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:47.261660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.261660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:47.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:47.268732 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:47.268910 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:47.269081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:26:47.269634 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.269955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.270590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:47.270590 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:47.270590 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:47.270590 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:47.270590 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:47.270590 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:47.270590 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.270590 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:47.271603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.271603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:47.271603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:47.271603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:47.271603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.271603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:47.271603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:47.271603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:47.271603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.271603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:47.271603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:47.271603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:47.271603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:47.271603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:47.271603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:47.271876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:47.271876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:47.271876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:47.271876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:47.271876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:47.271876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:47.271876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:47.271876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.271876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:47.272622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.272622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:47.272622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:47.272622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:47.272622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.272622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:47.272622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:47.272622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:26:47.272622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:47.272622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:47.272622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.272622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:47.272947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:47.272947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:47.272947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.272947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:47.272947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:47.272947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:47.272947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:47.272947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:47.272947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:47.272947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:47.272947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:47.272947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:47.272947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:47.272947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:47.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:47.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:47.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:47.273779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.273779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:47.273779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:47.273779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:47.273779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:47.273779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:47.273779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.273779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:47.273779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:47.273779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.273779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:47.273779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:47.273779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.273779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:47.273779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:47.273779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:47.274225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:47.274604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:47.274604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:47.274604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:47.274604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:47.274604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:47.274604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.274604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:47.276205 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:47.276509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:47.276689 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:47.276873 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:47.277375 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.277718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.278335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:47.278335 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:47.278335 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:47.278335 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:47.278335 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:47.278335 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:47.278335 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.278335 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:47.279193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.279193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:47.279193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:47.279193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:47.279193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.279193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:47.279193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:47.279193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:47.279193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.279193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:47.279193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:47.279193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:47.279193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:47.279193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:47.279193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:47.279495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:47.279495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:47.279495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:47.279495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:47.279495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:47.279495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:47.279495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:47.279495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.279495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:47.280110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.280110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:47.280110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:47.280110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:47.280110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.280110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:47.280110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:47.280110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:47.280110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:47.280110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.280110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:47.280338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:47.280338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:47.280338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.280338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:47.280338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:47.280338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:47.280338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:47.280338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:47.280338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:47.280338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:47.280338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:47.280338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:47.280338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:47.280338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:47.280694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:47.280694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:47.280694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.280694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.281185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:47.281492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:47.281958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:47.281958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:47.281958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.281958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:47.281958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:47.281958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:47.281958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:47.281958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:47.282453 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.282787 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:47.283150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.283716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:47.283716 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:47.283716 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:47.283716 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:47.283716 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:47.283716 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:47.283716 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.283716 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:47.284696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.284696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:47.284696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:47.284696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:47.284696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.284696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:47.284696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:47.284696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.284696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:47.284696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:47.284696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:47.284696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.284942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:47.284942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:47.284942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:47.284942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:47.284942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:47.284942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:47.284942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:47.284942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:47.284942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:47.284942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:47.284942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:47.284942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:47.284942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:47.285159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.285159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:47.285759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.285759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:47.285759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:47.285759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:47.285759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:47.285759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:47.285759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.285759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:47.285759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:47.285759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.285759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:47.285759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:47.285759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.285759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:47.285759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:47.285759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:47.286130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:47.286473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:47.286473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:47.286473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:51.800980 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:51.801463 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:51.801609 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:51.802353 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:51.865568 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:51.865848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:51.866725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:51.866725 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:51.866725 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:51.866725 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:51.866725 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:51.866725 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:51.866725 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:51.866725 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:51.867839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.065 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:51.867839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:51.867839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:51.867839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:51.867839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:51.867839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:51.867839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:51.867839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:51.867839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:51.867839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:51.867839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:51.867839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:51.868335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:51.868335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:51.868335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:51.868335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:51.868335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:51.868335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:51.868335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:51.868335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:51.868335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:51.868335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:51.868335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:51.868335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:51.868335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:51.868838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:51.868838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:51.868922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:51.868922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:51.868922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:51.868922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:51.868922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:51.868922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:51.868922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:51.868922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:51.868922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:51.868922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:51.868922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:51.868922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:51.868922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:51.868922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:51.868922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:51.868922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:51.869663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:51.870167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:51.870167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:51.870167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:52.254700 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:52.255309 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:52.255429 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:52.256809 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.257019 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.257191 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.257333 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.257550 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.259443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.259443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.259443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.259443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.259443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-714aff1c-85de-462e-a525-1bfe2ba61b53, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-01b0936e-ffbb-4414-80a1-4711a57d63a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.259858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-49504799-a156-4a0d-8b77-bb5f003180c1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.260092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=15): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.260092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=16): DelStaticRoutesCommand(_result=None, lrouter=neutron-6dd139ce-2ca3-470e-b5db-f6d77d05e04c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.260092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.260209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:52.260209 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:52.260209 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:52.260209 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:52.260209 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:52.260209 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:52.260209 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.260209 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:52.262702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.262702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:52.262702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:52.262702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:52.262702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.262702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:52.262702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:52.262702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.262702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:52.262702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:52.262702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:52.262702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.263619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:52.263619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:52.263619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:52.263619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:52.263619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:52.263619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:52.263619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:52.263619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:52.263619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:52.263619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:52.263619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:52.263619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:52.263619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:52.264282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.264282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:52.264282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.264282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:52.264282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:52.264282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:52.264282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:52.264282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:52.264282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.264282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:52.264282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:52.264282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.264282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:52.264282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:52.264282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:52.264821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:52.265377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:52.265377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:52.265377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:52.265377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:52.265377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.265377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:52.269075 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:52.269550 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:52.269706 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:52.269954 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:26:52.270943 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.271272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.272265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:52.272265 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:52.272265 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:52.272265 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:52.272265 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:52.272265 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:52.272265 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.272265 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:52.273298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.273298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:52.273298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:52.273298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:52.273298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.273298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:52.273298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:52.273298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:52.273298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.273298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:52.273298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:52.273298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:52.273298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:52.273298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:52.273298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:52.273876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:52.273876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:52.273876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:52.273876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:52.273876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:52.273876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:52.273876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:52.273876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.273876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:52.274353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.274353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:52.274353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:52.274353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:52.274353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.274353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:52.274353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:52.274353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:26:52.274353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:52.274353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:52.274353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.274353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:52.275117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:52.275117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:52.275117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.275117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:52.275117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:52.275117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:52.275117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:52.275117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:52.275117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:52.275117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:52.275117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:52.275117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:52.275117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:52.275117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:52.275793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:52.275793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:52.275793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.275793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:52.275793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.275793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:52.275793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:52.275793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:52.275793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:52.275793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:52.275793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.275793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:52.275793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:52.275793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:52.276225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:52.276686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:52.276686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:52.276686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:52.276686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:52.276686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:52.276686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:52.276686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:52.276686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:52.276686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:52.276686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:52.276686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.276686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:52.276686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:52.277041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:52.277041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:52.277041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:52.279711 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.279711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.279711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:52.279711 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:52.279711 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:52.279711 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:52.279711 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:52.279711 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:52.279711 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.279711 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:52.283734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.283734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:52.283734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:52.283734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:52.283734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.283734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:52.283734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:52.283734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:52.283734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.283734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:52.283734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:52.283734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:52.283734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:52.283734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:52.283734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:52.285866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:52.285866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:52.285866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:52.285866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:52.285866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:52.285866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:52.285866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:52.285866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.285866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:52.286278 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:52.288489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.288489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:52.288489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:52.288489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:52.288489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.288489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:52.288489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:52.288489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:52.288489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:52.288489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.288489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:52.290166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:52.290166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:52.290166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.290166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:52.290166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:52.290166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:52.290166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:52.290166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:52.290166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:52.290166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:52.290166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:52.290166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:52.290166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:52.290166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:52.290899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:52.290899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:52.290899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.290899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:52.290899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.290899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:52.290899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:52.290899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:52.290899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:52.290899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:52.290899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.290899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:52.290899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:52.290899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:52.291979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:52.292871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:52.292871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:52.292871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:52.292871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:52.292871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:52.292871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:52.292871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:52.292871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:52.292871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:52.292871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.292871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:52.292871 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:52.292871 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:52.293241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:52.293241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.293241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:52.293241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.293241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:52.293241 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:52.293241 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:52.293241 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:52.293241 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:52.293241 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:52.293241 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.293241 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:52.293741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.293741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:52.293741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:52.293741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:52.293741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.293741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:52.293741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:52.293741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.293741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:52.293741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:52.293741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:52.293741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.294889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:52.294889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:52.294889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:52.294889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:52.294889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:52.294889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:52.294889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:52.294889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:52.294889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:52.294889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:52.294889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:52.294889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:52.294889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:52.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:52.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:52.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:52.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:52.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:52.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:52.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:52.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:52.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:52.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:52.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:52.295938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:52.296392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:52.296392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:52.296392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:52.296392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:52.296392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:52.296392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:56.804659 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:56.806592 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:56.806592 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:56.806735 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:56.856120 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:56.856263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:56.856889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:56.856889 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:56.856889 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:56.856889 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:56.856889 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:56.856889 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:56.856889 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:56.856889 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:56.857792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.051 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:56.857792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:56.857792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:56.857792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:56.857792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:56.857792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:56.857792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:56.857792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:56.857792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:56.857792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:56.857792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:56.857792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:56.858168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:56.858168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:56.858168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:56.858168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:56.858168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:56.858168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:56.858168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:56.858168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:56.858168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:56.858168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:56.858168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:56.858168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:56.858168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:56.858559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:56.858559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:56.858899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:56.858899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:56.858899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:56.858899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:56.858899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:56.858899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:56.858899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:56.858899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:56.858899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:56.858899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:56.858899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:56.858899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:56.858899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:56.858899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:56.858899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:56.858899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:56.859270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:56.859613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:56.859613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:56.859613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:57.255437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:57.255740 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:57.255881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:57.257492 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.257703 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.257843 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.258000 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.258164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.258301 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.258470 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.258634 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.258774 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.258902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.259037 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.259167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.259289 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-01b0936e-ffbb-4414-80a1-4711a57d63a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.259408 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-49504799-a156-4a0d-8b77-bb5f003180c1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.259567 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.259686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=15): DelStaticRoutesCommand(_result=None, lrouter=neutron-6dd139ce-2ca3-470e-b5db-f6d77d05e04c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.259822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.260368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:57.260368 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:57.260368 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:57.260368 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:57.260368 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:57.260368 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:57.260368 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.260368 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:57.261310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.261310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:57.261310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:57.261310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:57.261310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.261310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:57.261310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:57.261310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.261310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:57.261310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:57.261310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:57.261310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.263593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:57.263593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:57.263593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:57.263593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:57.263593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:57.263593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:57.263593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:57.263593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:57.263593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:57.263593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:57.263593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:57.263593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:57.263593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:57.263840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.263840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:57.263840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.263840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:57.263840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:57.263840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:57.263840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:57.263840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:57.263840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.263840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:57.263840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:57.263840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.263840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:57.263840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:57.263840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:57.264089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:57.264414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:57.264414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:57.264414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:57.264414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:57.264414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.264414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:57.269559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:57.269852 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:57.269970 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:57.270102 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:26:57.270501 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.270709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.271515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:57.271515 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:57.271515 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:57.271515 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:57.271515 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:57.271515 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:57.271515 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.271515 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:57.272404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.272404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:57.272404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:57.272404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:57.272404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.272404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:57.272404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:57.272404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:57.272404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.272404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:57.272404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:57.272404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:57.272404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:57.272404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:57.272404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:57.272730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:57.272730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:57.272730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:57.272730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:57.272730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:57.272730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:57.272730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:57.272730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.272730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:57.273371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.273371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:57.273371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:57.273371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:57.273371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.273371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:57.273371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:57.273371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:26:57.273371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:57.273371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:57.273371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.273371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:57.273677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:57.273677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:57.273677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.273677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:57.273677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:57.273677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:57.273677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:57.273677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:57.273677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:57.273677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:57.273677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:57.273677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:57.273677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:57.273677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:57.273911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:57.273911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:57.273911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.273911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:57.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:57.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:57.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:57.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:57.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:57.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:57.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:57.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:57.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:57.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:57.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:57.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:57.274776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:57.275092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:57.275092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:57.275092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:57.275092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:57.275092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:57.275092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.275092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:57.277147 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:57.277381 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:57.277511 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:57.277661 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:57.278003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.278197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.278705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:57.278705 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:57.278705 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:57.278705 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:57.278705 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:57.278705 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:57.278705 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.278705 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:57.279492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.279492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:26:57.279492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:57.279492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:57.279492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.279492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:57.279492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:57.279492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:26:57.279492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.279492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:26:57.279492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:57.279492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:26:57.279492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:57.279492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:26:57.279492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:26:57.279944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:57.279944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:26:57.279944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:57.279944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:26:57.279944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:26:57.279944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:57.279944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:26:57.279944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.279944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:26:57.280432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.280432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:57.280432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:57.280432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:57.280432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.280432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:57.280432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:57.280432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:57.280432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:57.280432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.280432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:57.280861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:57.280861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:57.280861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.280861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:57.280861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:57.280861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:57.280861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:57.280861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:57.280861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:57.280861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:57.280861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:57.280861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:57.280861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:57.280861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:57.281349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:57.281349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:57.281349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.281349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.281669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:57.282212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:57.282990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:57.282990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:57.282990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.282990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:26:57.289670 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:26:57.290116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:26:57.290289 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:26:57.290532 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:26:57.291096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.291325 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:26:57.291805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.292634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:26:57.292634 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:57.292634 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:26:57.292634 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:26:57.292634 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:57.292634 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:26:57.292634 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.292634 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:26:57.293701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.293701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:26:57.293701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:57.293701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:26:57.293701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.293701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:57.293701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:57.293701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.293701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:57.293701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:57.293701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:26:57.293701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.294518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:26:57.294518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:57.294518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:26:57.294518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:57.294518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:26:57.294518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:26:57.294518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:57.294518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:26:57.294518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:57.294518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:26:57.294518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:26:57.294518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:57.294518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:26:57.294879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.294879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:26:57.294879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.294879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:26:57.294879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:26:57.294879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:26:57.294879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:26:57.294879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:26:57.294879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.294879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:26:57.294879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:26:57.294879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.294879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:26:57.294879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:26:57.294879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:26:57.295405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:26:57.295917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:26:57.295917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:26:57.295917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:26:57.295917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:26:57.295917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:26:57.295917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:01.807955 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:01.807955 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:01.807955 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:01.808832 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:01.906699 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:01.907881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:01.910422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:01.910422 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:01.910422 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:01.910422 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:01.910422 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:01.910422 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:01.910422 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:01.910422 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:01.911479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.103 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:01.911479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:01.911479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:01.911479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:01.911479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:01.911479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:01.911479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:01.911479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:01.911479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:01.911479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:01.911479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:01.911479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:01.912156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:01.912156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:01.912156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:01.912156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:01.912156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:01.912156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:01.912156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:01.912156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:01.912156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:01.912156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:01.912156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:01.912156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:01.912156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:01.912752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:01.912752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:01.912752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:01.912752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:01.912752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:01.912752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:01.912752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:01.912752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:01.912752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:01.912752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:01.912752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:01.912752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:01.912752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:01.912752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:01.912752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:01.914183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:01.915188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:01.915188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:01.915188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:01.915188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:01.915188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:01.915188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:02.256268 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:02.256844 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:02.256928 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:02.258318 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.258627 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.258768 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.258930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.259066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.259213 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.259352 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.260394 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.260394 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.260394 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.260394 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.260394 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.260722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-01b0936e-ffbb-4414-80a1-4711a57d63a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.260722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.260780 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-6dd139ce-2ca3-470e-b5db-f6d77d05e04c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.260864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.261755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:02.261755 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:02.261755 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:02.261755 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:02.261755 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:02.261755 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:02.261755 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.261755 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:02.262858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.262858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:02.262858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:02.262858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:02.262858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.262858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:02.262858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:02.262858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.262858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:02.262858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:02.262858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:02.262858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.263140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:02.263140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:02.263140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:02.263140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:02.263140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:02.263140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:02.263140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:02.263140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:02.263140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:02.263140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:02.263140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:02.263140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:02.263140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:02.263379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.263379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:02.263982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.263982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:02.263982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:02.263982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:02.263982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:02.263982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:02.263982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.263982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:02.263982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:02.263982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.263982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:02.263982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:02.263982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.263982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:02.263982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:02.263982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:02.264658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:02.265361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:02.265361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.265361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:02.270596 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:02.271084 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:02.271206 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:02.271343 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:27:02.272364 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.273086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.273153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:02.273153 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:02.273153 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:02.273153 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:02.273153 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:02.273153 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:02.273153 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.273153 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:02.273874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.273874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:02.273874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:02.273874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:02.273874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.273874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:02.273874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:02.273874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:02.273874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.273874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:02.273874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:02.273874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:02.273874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:02.273874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:02.273874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:02.274281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:02.274281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:02.274281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:02.274281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:02.274281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:02.274281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:02.274281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:02.274281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.274281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:02.274607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.274607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:02.274607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:02.274607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:02.274607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.274607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:02.274607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:02.274607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:27:02.274607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:02.274607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:02.274607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.274607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:02.274907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:02.274907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:02.274907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.274907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:02.274907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:02.274907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:02.274907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:02.274907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:02.274907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:02.274907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:02.274907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:02.274907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:02.274907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:02.274907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:02.275222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:02.275222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:02.275222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.275222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:02.275222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.275222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:02.275222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:02.275222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:02.275222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:02.275222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:02.275222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.275222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:02.275222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:02.275222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:02.275568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:02.275989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:02.275989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:02.275989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:02.275989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:02.275989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:02.275989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:02.275989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:02.275989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:02.275989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:02.275989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:02.275989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.275989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:02.277520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:02.277879 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:02.278004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:02.278147 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:02.278592 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.278836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.279697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:02.279697 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:02.279697 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:02.279697 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:02.279697 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:02.279697 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:02.279697 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.279697 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:02.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:02.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:02.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:02.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:02.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:02.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:02.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:02.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:02.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:02.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:02.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:02.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:02.280989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:02.280989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:02.280989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:02.280989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:02.280989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:02.280989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:02.280989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:02.280989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.280989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:02.281658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.281658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:02.281658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:02.281658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:02.281658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.281658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:02.281658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:02.281658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:02.281658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:02.281658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.281658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:02.281948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:02.281948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:02.281948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.281948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:02.281948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:02.281948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:02.281948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:02.281948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:02.281948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:02.281948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:02.281948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:02.281948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:02.281948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:02.281948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:02.282271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:02.282271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:02.282271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.282271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:02.283130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:02.283469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:02.283469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:02.283469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.283469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:02.293568 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:02.293568 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:02.293568 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:02.293568 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:02.293568 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.293568 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:02.293892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.293892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:02.293892 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:02.293892 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:02.293892 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:02.293892 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:02.293892 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:02.293892 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.293892 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:02.293892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.293892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:02.293892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:02.293892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:02.293892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.293892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:02.294320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:02.294320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.294320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:02.294320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:02.294320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:02.294320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.294320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:02.294320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:02.294320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:02.294320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:02.294320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:02.294320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:02.294320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:02.294320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:02.294570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:02.294570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:02.294570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:02.294570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:02.294570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:02.294570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.294570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:02.294943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.294943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:02.294943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:02.294943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:02.294943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:02.294943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:02.294943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.294943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:02.294943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:02.294943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.294943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:02.294943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:02.294943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.294943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:02.294943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:02.294943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:02.295276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:02.295642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:02.295642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:02.295642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:06.807662 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:06.808223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:06.808358 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:06.809246 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:06.865279 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:06.866048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:06.866200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:06.866200 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:06.866200 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:06.866200 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:06.866200 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:06.866200 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:06.866200 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:06.866200 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:06.866848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:06.866848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:06.866848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:06.866848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:06.866848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:06.866848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:06.866848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:06.866848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:06.866848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:06.866848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:06.866848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:06.866848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:06.867193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:06.867193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:06.867193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:06.867193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:06.867193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:06.867193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:06.867193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:06.867193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:06.867193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:06.867193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:06.867193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:06.867193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:06.867193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:06.867510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:06.867510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:06.867587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:06.867587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:06.867587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:06.867587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:06.867587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:06.867587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:06.867587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:06.867587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:06.867587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:06.867587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:06.867587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:06.867587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:06.867587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:06.867587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:06.867587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:06.867587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:06.867976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:06.868381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:06.868381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:06.868381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:07.258321 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:07.258770 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:07.258871 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:07.260671 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.260747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.260939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.261603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.261603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.261603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.261759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.261914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.262076 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.262233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.262415 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.262605 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.262761 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-01b0936e-ffbb-4414-80a1-4711a57d63a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.262918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.263075 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-6dd139ce-2ca3-470e-b5db-f6d77d05e04c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.263229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:07.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:07.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:07.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:07.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:07.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:07.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:07.266897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.266897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:07.266897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:07.266897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:07.266897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.266897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:07.266897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:07.266897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.266897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:07.266897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:07.266897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:07.266897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.267402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:07.267402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:07.267402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:07.267402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:07.267402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:07.267402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:07.267402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:07.267402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:07.267402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:07.267402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:07.267402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:07.267402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:07.267402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:07.267792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.267792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:07.268168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.268168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:07.268168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:07.268168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:07.268168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:07.268168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:07.268168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.268168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:07.268168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:07.268168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.268168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:07.268168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:07.268168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.268168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:07.268168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:07.268168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:07.268648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:07.269167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:07.269167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.269167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:07.272578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:07.272578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:07.272578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:07.272578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:27:07.272956 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.273249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.274744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:07.274744 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:07.274744 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:07.274744 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:07.274744 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:07.274744 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:07.274744 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.274744 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:07.276723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:07.277239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:07.277239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:07.277239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:07.277239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:07.277239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:07.277239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:07.277239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:07.277239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.277239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:07.277580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.277580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:07.277580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:07.277580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:07.277580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.277580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:07.277580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:07.277580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:27:07.277580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:07.277580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:07.277580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.277580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:07.278100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:07.278100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:07.278100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.278100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:07.278100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:07.278100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:07.278100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:07.278100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:07.278100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:07.278100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:07.278100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:07.278100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:07.278100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:07.278100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:07.278593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:07.278593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:07.278593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.278593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:07.279191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.279191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:07.279191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:07.279191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:07.279191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:07.279191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:07.279191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.279191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:07.279191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:07.279191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.279191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:07.279191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:07.279191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.279191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:07.279191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:07.279191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:07.279721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:07.280168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:07.280168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:07.280168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:07.280168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:07.280168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:07.280168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.280168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:07.280168 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:07.280168 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:07.280168 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:07.280464 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:07.280963 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.281229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.281869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:07.281869 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:07.281869 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:07.281869 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:07.281869 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:07.281869 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:07.281869 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.281869 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:07.283456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.283456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:07.283456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:07.283456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:07.283456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.283456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:07.283456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:07.283456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:07.283456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.283456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:07.283456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:07.283456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:07.283456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:07.283456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:07.283456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:07.284113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:07.284113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:07.284113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:07.284113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:07.284113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:07.284113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:07.284113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:07.284113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.284113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:07.284466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.284466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:07.284466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:07.284466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:07.284466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.284466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:07.284466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:07.284466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:07.284466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:07.284466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.284466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:07.285148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:07.285148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:07.285148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.285148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:07.285148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:07.285148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:07.285148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:07.285148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:07.285148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:07.285148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:07.285148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:07.285148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:07.285148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:07.285148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:07.285673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:07.285673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:07.285673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.285673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.285975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:07.287027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:07.287881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:07.287881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:07.287881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.287881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:07.290790 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:07.291146 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:07.291286 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:07.291487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:07.292670 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.292865 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:07.293186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.293930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:07.293930 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:07.293930 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:07.293930 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:07.293930 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:07.293930 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:07.293930 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.293930 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:07.296691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.296691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:07.296691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:07.296691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:07.296691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.296691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:07.296691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:07.296691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.296691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:07.296691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:07.296691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:07.296691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.297164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:07.297164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:07.297164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:07.297164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:07.297164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:07.297164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:07.297164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:07.297164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:07.297164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:07.297164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:07.297164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:07.297164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:07.297164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:07.297597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.297597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:07.299995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.299995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:07.299995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:07.299995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:07.299995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:07.299995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:07.299995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.299995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:07.299995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:07.299995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.299995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:07.299995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:07.299995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.299995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:07.299995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:07.299995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:07.300486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:07.301238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:07.301238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:07.301238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:11.808904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:11.811594 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:11.811594 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:11.811594 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:11.889307 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:11.889307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:11.889600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:11.889600 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:11.889600 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:11.889600 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:11.889600 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:11.889600 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:11.889600 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:11.889600 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:11.890778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.080 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:11.890778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:11.890778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:11.890778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:11.890778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:11.890778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:11.890778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:11.890778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:11.890778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:11.890778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:11.890778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:11.890778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:11.891179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:11.891179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:11.891179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:11.891179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:11.891179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:11.891179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:11.891179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:11.891179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:11.891179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:11.891179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:11.891179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:11.891179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:11.891179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:11.891503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:11.891503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:11.891503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:11.891503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:11.891503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:11.891503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:11.891503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:11.891503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:11.891503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:11.891503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:11.891503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:11.891503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:11.891503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:11.891503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:11.891503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:11.891908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:11.892380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:11.892380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:11.892380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:11.892380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:11.892380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:11.892380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:12.258854 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:12.259968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:12.262402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:12.262402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.262687 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.262856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.263009 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.263301 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.263301 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.263442 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.263577 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.263728 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.263876 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.264013 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.264259 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.264334 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-01b0936e-ffbb-4414-80a1-4711a57d63a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.264426 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.264601 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-6dd139ce-2ca3-470e-b5db-f6d77d05e04c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.264747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=15): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.264879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.265633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:12.265633 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:12.265633 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:12.265633 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:12.265633 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:12.265633 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:12.265633 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.265633 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:12.266344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.266344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:12.266344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:12.266344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:12.266344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.266344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:12.266344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:12.266344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.266344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:12.266344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:12.266344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:12.266344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.266779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:12.266779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:12.266779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:12.266779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:12.266779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:12.266779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:12.266779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:12.266779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:12.266779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:12.266779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:12.266779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:12.266779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:12.266779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:12.267013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.267013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:12.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:12.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:12.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:12.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:12.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:12.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:12.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:12.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:12.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:12.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:12.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:12.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:12.268476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:12.269298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:12.269298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.269298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:12.272013 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:12.272525 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:12.272651 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:12.272813 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:27:12.276253 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.276253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.276253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:12.276253 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:12.276253 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:12.276253 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:12.276253 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:12.276253 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:12.276253 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.276253 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:12.276253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.276253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:12.276253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:12.276253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:12.276979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:12.277499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.277499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:12.277499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.277499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:12.277499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:12.277499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:12.277499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.277499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:12.277499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:12.277499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:27:12.277499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:12.277499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:12.277907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.277907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:12.277907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:12.277907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:12.277907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.277907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:12.277907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:12.277907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:12.277907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:12.277907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:12.277907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:12.277907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:12.277907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:12.277907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:12.278416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:12.278416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:12.278416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:12.278416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:12.278416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.278416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:12.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:12.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:12.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:12.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:12.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:12.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:12.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:12.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:12.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:12.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:12.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:12.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:12.279995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:12.280705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:12.280705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:12.280705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:12.280705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:12.280705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:12.280705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.280705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:12.280705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:12.280705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:12.280705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:12.281023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:12.283282 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.283282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.283282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:12.283282 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:12.283282 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:12.283282 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:12.283282 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:12.283282 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:12.283282 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.283282 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:12.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:12.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:12.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:12.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:12.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:12.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:12.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:12.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:12.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:12.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:12.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:12.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:12.284378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:12.284378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:12.284378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:12.284378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:12.284378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:12.284378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:12.284378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:12.284378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.284378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:12.284968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.284968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:12.284968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:12.284968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:12.284968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.284968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:12.284968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:12.284968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:12.284968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:12.284968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.284968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:12.285289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:12.285289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:12.285289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.285289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:12.285289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:12.285289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:12.285289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:12.285289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:12.285289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:12.285289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:12.285289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:12.285289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:12.285289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:12.285289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:12.285683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:12.285683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:12.285683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.285683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.286180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:12.286680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:12.287128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:12.287128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:12.287128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.287128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:12.291767 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:12.292192 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:12.292266 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:12.292424 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:12.292995 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.293464 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:12.294128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.295793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:12.295793 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:12.295793 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:12.295793 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:12.295793 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:12.295793 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:12.295793 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.295793 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:12.296999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.296999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:12.296999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:12.296999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:12.296999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.296999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:12.296999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:12.296999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.296999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:12.296999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:12.296999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:12.296999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.297410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:12.297410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:12.297410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:12.297410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:12.297410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:12.297410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:12.297410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:12.297410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:12.297410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:12.297410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:12.297410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:12.297410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:12.297410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:12.297798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.297798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:12.297798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.297798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:12.297798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:12.297798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:12.297798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:12.297798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:12.297798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.297798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:12.297798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:12.297798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.297798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:12.297798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:12.297798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:12.298463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:12.299166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:12.299166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:12.299166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:12.299166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:12.299166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:12.299166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:16.811420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:16.812011 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:16.812011 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:16.812657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:16.900884 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:16.901148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:16.902722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:16.902722 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:16.902722 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:16.902722 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:16.902722 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:16.902722 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:16.902722 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:16.902722 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:16.903669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.091 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:16.903669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:16.903669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:16.903669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:16.903669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:16.903669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:16.903669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:16.903669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:16.903669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:16.903669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:16.903669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:16.903669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:16.904318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:16.904318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:16.904318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:16.904318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:16.904318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:16.904318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:16.904318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:16.904318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:16.904318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:16.904318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:16.904318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:16.904318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:16.904318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:16.904693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:16.904693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:16.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:16.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:16.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:16.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:16.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:16.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:16.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:16.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:16.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:16.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:16.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:16.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:16.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:16.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:16.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:16.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:16.905195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:16.909967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:16.909967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:16.909967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:17.260058 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:17.260379 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:17.260512 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:17.262004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.262311 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.262402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.262499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.262657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.262745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.262823 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.262906 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.262983 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-01b0936e-ffbb-4414-80a1-4711a57d63a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.264151 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-6dd139ce-2ca3-470e-b5db-f6d77d05e04c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.264151 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=15): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.264151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.264717 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:17.265255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.265255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:17.265255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:17.265255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:17.265255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.265255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:17.265255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:17.265255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.265255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:17.265255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:17.265255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:17.265255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.266982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:17.266982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:17.266982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:17.266982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:17.266982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:17.266982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:17.266982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:17.266982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:17.266982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:17.266982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:17.266982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:17.266982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:17.266982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:17.267817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.267817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:17.267817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.267817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:17.267817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:17.267817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:17.267817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:17.267817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:17.267817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.267817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:17.267817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:17.267817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.267817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:17.267817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:17.267817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:17.268329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:17.269806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:17.269806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:17.269806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:17.269806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:17.269806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.269806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:17.272953 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:17.279399 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:17.279660 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:17.279831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:27:17.280531 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.281409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.283156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:17.283156 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:17.283156 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:17.283156 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:17.283156 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:17.283156 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:17.283156 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.283156 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:17.283723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:17.284290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.284290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:17.284290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:17.284290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:17.284290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.284290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:17.284290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:17.284290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:17.284290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.284290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:17.284290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:17.284290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:17.284290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:17.284290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:17.284290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:17.285381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:17.285381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:17.285381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:17.285381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:17.285381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:17.285381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:17.285381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:17.285381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.285381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:17.285381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.285381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:17.285381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:17.285381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:17.285381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.286187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:17.286187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:17.286187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:27:17.286187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:17.286187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:17.286187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.286187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:17.286187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:17.286187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:17.286187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.286187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:17.286187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:17.286187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:17.286631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:17.286631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:17.286631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:17.286631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:17.286631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:17.286631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:17.286631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:17.286631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:17.286631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:17.286631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:17.286631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.286631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:17.286970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.286970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:17.286970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:17.286970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:17.286970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:17.286970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:17.286970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.286970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:17.286970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:17.286970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.286970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:17.286970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:17.286970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.286970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:17.286970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:17.286970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:17.287485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:17.289492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:17.289492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:17.289492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:17.289492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:17.289492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:17.289492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.289492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:17.289492 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:17.289492 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:17.289492 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:17.290139 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.290139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.290139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:17.290139 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:17.290139 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:17.290139 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:17.290139 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:17.290139 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:17.290139 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.290139 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:17.290139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.290139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:17.290139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:17.290139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:17.291726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:17.292208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.292208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:17.292208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.292208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:17.292208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:17.292208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:17.292208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.292208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:17.292208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:17.292208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:17.292208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:17.292208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.292598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:17.292598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:17.292598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:17.292598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.292598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:17.292598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:17.292598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:17.292598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:17.292598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:17.292598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:17.292598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:17.292598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:17.292598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:17.292598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:17.292986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:17.292986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:17.292986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:17.292986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.292986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:17.292986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.292986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:17.292986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:17.292986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:17.292986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:17.292986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:17.292986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.292986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:17.292986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:17.292986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:17.293408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:17.293962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:17.293962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:17.293962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:17.293962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:17.293962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:17.293962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:17.293962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:17.293962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:17.293962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:17.293962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.293962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:17.293962 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:17.293962 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:17.294329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:17.294329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:17.294329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.294329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:17.294329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.294329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:17.294329 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:17.294329 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:17.294329 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:17.296155 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:17.296155 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:17.296155 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.296155 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:17.296155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.296155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:17.296155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:17.296155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:17.296155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.296155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:17.296155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:17.296155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.296155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:17.296155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:17.297724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:17.297724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.297724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:17.297724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:17.297724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:17.297724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:17.297724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:17.297724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:17.297724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:17.297724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:17.297724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:17.297724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:17.297724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:17.297724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:17.298533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:17.298533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.298533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:17.298533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.298533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:17.298533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:17.298533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:17.298533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:17.298533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:17.298533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.298533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:17.298533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:17.298533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.298533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:17.298533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:17.298533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:17.300069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:17.300497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:17.300497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:17.300497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:17.300497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:17.300497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:17.300497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:21.811982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:21.812828 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:21.812828 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:21.814746 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:21.820265 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=240 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:27:21.898382 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:21.898382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:21.899416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:21.899416 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:21.899416 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:21.899416 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:21.899416 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:21.899416 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:21.899416 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:21.899416 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:21.901092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.087 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:21.901092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:21.901092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:21.901092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:21.901092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:21.901092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:21.901092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:21.901092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:21.901092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:21.901092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:21.901092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:21.901092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:21.901610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:21.901610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:21.901610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:21.901610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:21.901610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:21.901610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:21.901610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:21.901610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:21.901610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:21.901610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:21.901610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:21.901610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:21.901610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:21.902060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:21.902060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:21.902060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:21.902060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:21.902060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:21.902060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:21.902060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:21.902060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:21.902060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:21.902060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:21.902060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:21.902060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:21.902060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:21.902060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:21.902060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:21.902512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:21.903012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:21.903012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:21.903012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:21.903012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:21.903012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:21.903012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:22.262363 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:22.264055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:22.264843 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:22.269086 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.270016 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.270575 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.272690 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.272690 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.272690 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.272690 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.272690 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.272988 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.272988 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.272988 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-01b0936e-ffbb-4414-80a1-4711a57d63a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.272988 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.273409 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-6dd139ce-2ca3-470e-b5db-f6d77d05e04c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.273409 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.273409 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.273769 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=15): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.274471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.275075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:22.275075 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:22.275075 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:22.275075 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:22.275075 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:22.275075 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:22.275075 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.275075 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:22.276086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.011 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.276086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:22.276086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:22.276086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:22.276086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.276086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:22.276086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:22.276086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.276086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:22.276086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:22.276086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:22.276086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.276916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:22.276916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:22.276916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:22.276916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:22.276916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:22.276916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:22.276916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:22.276916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:22.276916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:22.276916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:22.276916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:22.276916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:22.276916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:22.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:22.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:22.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:22.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:22.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:22.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:22.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:22.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:22.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:22.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:22.278361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:22.278936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:22.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:22.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:22.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:22.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:22.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:22.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:22.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:22.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:22.279446 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:27:22.280974 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.280974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.280974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:22.280974 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:22.280974 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:22.280974 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:22.280974 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:22.280974 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:22.280974 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.280974 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:22.281311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.281311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:22.281311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:22.281311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:22.281311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.281311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:22.281311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:22.281311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:22.281311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.281311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:22.281311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:22.281311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:22.281311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:22.281311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:22.281311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:22.284206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:22.284206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:22.284206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:22.284206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:22.284206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:22.284206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:22.284206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:22.284206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.284206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:22.284206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.284206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:22.284206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:22.284206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:22.284206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:22.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:22.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:27:22.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:22.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:22.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:22.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:22.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:22.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:22.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:22.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:22.285066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:22.285066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:22.285066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:22.285066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:22.285066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:22.285066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:22.285066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:22.285066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:22.285066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:22.285066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:22.285066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.285066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:22.285357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.285357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:22.285357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:22.285357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:22.285357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:22.285357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:22.285357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.285357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:22.285357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:22.285357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.285357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:22.285357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:22.285357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.285357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:22.285357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:22.285357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:22.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:22.286118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:22.286118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:22.286118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:22.286118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:22.286118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:22.286118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.286118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:22.287600 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:22.288045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:22.288045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:22.288045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:22.288511 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.289048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.290080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:22.290080 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:22.290080 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:22.290080 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:22.290080 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:22.290080 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:22.290080 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.290080 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:22.290461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.290461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:22.290461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:22.290461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:22.290461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.290461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:22.290461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:22.290461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:22.290461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.290461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:22.290461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:22.290461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:22.290461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:22.290461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:22.290461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:22.290844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:22.290844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:22.290844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:22.290844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:22.290844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:22.290844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:22.290844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:22.290844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.290844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:22.291123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.291123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:22.291123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:22.291123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:22.291123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.291123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:22.291123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:22.291123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:22.291123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:22.291123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.291123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:22.291325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:22.291325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:22.291325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.291325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:22.291325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:22.291325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:22.291325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:22.291325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:22.291325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:22.291325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:22.291325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:22.291325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:22.291325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:22.291325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:22.291613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:22.291613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:22.291613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.291613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.292203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:22.293203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:22.293565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:22.293565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:22.293565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.293565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:22.293565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:22.293565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:22.293565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:22.293565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:22.293920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.294303 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:22.294941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.295515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:22.295515 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:22.295515 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:22.295515 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:22.295515 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:22.295515 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:22.295515 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.295515 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:22.296205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.296205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:22.296205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:22.296205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:22.296205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.296205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:22.296205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:22.296205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.296205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:22.296205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:22.296205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:22.296205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.296745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:22.296745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:22.296745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:22.296745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:22.296745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:22.296745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:22.296745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:22.296745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:22.296745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:22.296745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:22.296745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:22.296745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:22.296745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:22.297237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.297237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:22.297436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.297436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:22.297436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:22.297436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:22.297436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:22.297436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:22.297436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.297436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:22.297436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:22.297436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.297436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:22.297436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:22.297436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.297436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:22.297436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:22.297436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:22.297895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:22.298199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:22.298199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:22.298199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:26.812590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:26.813073 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:26.813204 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:26.813959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:26.870737 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:26.871001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:26.871675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:26.871675 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:26.871675 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:26.871675 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:26.871675 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:26.871675 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:26.871675 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:26.871675 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:26.872986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:26.872986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:26.872986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:26.872986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:26.872986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:26.872986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:26.872986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:26.872986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:26.872986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:26.872986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:26.872986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:26.872986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:26.873349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:26.873349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:26.873349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:26.873349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:26.873349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:26.873349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:26.873349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:26.873349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:26.873349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:26.873349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:26.873349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:26.873349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:26.873349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:26.873692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:26.873692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:26.874109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:26.874109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:26.874109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:26.874109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:26.874109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:26.874109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:26.874109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:26.874109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:26.874109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:26.874109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:26.874109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:26.874109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:26.874109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:26.874109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:26.874109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:26.874109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:26.874522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:26.875012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:26.875012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:26.875012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:27.265848 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:27.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:27.268032 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:27.271055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.271368 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.271633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.271814 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.272010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.272247 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.272343 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.272414 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.272592 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.276688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.276688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-01b0936e-ffbb-4414-80a1-4711a57d63a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.276688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.276688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-6dd139ce-2ca3-470e-b5db-f6d77d05e04c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.277098 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.277262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.277483 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=15): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.277741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.278422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:27.278422 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:27.278422 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:27.278422 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:27.278422 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:27.278422 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:27.278422 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.278422 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:27.279779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.279779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:27.279779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:27.279779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:27.279779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.279779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:27.279779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:27.279779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.279779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:27.279779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:27.279779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:27.279779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.280157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:27.280157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:27.280157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:27.280157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:27.280157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:27.280157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:27.280157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:27.280157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:27.280157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:27.280157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:27.280157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:27.280157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:27.280157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:27.280500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.280500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:27.282631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.282631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:27.282631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:27.282631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:27.282631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:27.282631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:27.282631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.282631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:27.282631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:27.282631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.282631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:27.282631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:27.282631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.282631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:27.282631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:27.282631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:27.283003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:27.283815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:27.283815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.283815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:27.283815 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:27.283815 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:27.283815 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:27.283815 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:27:27.283815 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:27.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:27.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:27.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:27.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:27.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:27.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.284016 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:27.285583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.285583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:27.285583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:27.285583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:27.285583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.285583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:27.285583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:27.285583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:27.285583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.285583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:27.285583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:27.285583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:27.285583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:27.285583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:27.285583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:27.286717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:27.286717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:27.286717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:27.286717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:27.286717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:27.286717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:27.286717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:27.286717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.286717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:27.286717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.286717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:27.286717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:27.286717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:27.286717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:27.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:27.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:27:27.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:27.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:27.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:27.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:27.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:27.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:27.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:27.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:27.287699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:27.287699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:27.287699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:27.287699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:27.287699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:27.287699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:27.287699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:27.287699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:27.287699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:27.287699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:27.287699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.287699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:27.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:27.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:27.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:27.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:27.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:27.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:27.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:27.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:27.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:27.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:27.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:27.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:27.288824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:27.293066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:27.293066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:27.293066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:27.293066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:27.293066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:27.293066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.293066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:27.293066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:27.293066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:27.293066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:27.293464 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:27.293464 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.293464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.293464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:27.293464 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:27.293464 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:27.293464 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:27.293464 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:27.293464 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:27.293464 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.293464 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:27.293464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:27.294348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:27.294886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:27.294886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:27.294886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:27.294886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:27.294886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.294886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:27.294886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.294886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:27.294886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:27.294886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:27.294886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.294886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:27.294886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:27.295312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:27.295312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:27.295312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.295312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:27.295312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:27.295312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:27.295312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.295312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:27.295312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:27.295312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:27.295312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:27.295312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:27.295312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:27.295801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:27.295801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:27.295801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:27.295801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:27.295801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:27.295801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:27.295801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:27.295801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.295801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:27.295801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.295801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:27.295801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.296194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.296739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:27.297412 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:27.297412 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:27.297412 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:27.297412 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:27.297412 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.297412 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:27.297717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.297717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:27.297717 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:27.297717 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:27.297717 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:27.297717 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:27.297717 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:27.297717 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.297717 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:27.297979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.297979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:27.297979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:27.297979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:27.297979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.297979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:27.297979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:27.297979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.297979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:27.297979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:27.297979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:27.297979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.298343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:27.298343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:27.298343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:27.298343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:27.298343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:27.298343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:27.298343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:27.298343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:27.298343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:27.298343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:27.298343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:27.298343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:27.298343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:27.298817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.298817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:27.298817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.298817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:27.298817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:27.298817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:27.298817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:27.298817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:27.298817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.298817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:27.298817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:27.298817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.298817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:27.298817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:27.298817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:27.299460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:27.300164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:27.300164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:27.300164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:27.300164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:27.300164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:27.300164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:31.813412 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:31.819621 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:31.819621 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:31.819621 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:31.899036 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:31.899312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:31.901078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:31.901078 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:31.901078 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:31.901078 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:31.901078 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:31.901078 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:31.901078 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:31.901078 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:31.903080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.084 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:31.903080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:31.903080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:31.903080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:31.903080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:31.903080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:31.903080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:31.903080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:31.903080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:31.903080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:31.903080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:31.903080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:31.903741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:31.903741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:31.903741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:31.903741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:31.903741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:31.903741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:31.903741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:31.903741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:31.903741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:31.903741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:31.903741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:31.903741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:31.903741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:31.904517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:31.904517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:31.904517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:31.904517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:31.904517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:31.904517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:31.904517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:31.904517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:31.904517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:31.904517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:31.904517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:31.904517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:31.904517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:31.904517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:31.904517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:31.904992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:31.905631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:31.905631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:31.905631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:31.905631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:31.905631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:31.905631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:32.267961 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:32.269059 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:32.269059 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:32.272339 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.272517 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-882cab9e-7a30-4db4-996b-4a5650918e4b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.272685 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.272810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.272917 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.273019 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.273459 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.273459 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.273459 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.273459 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.273915 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-01b0936e-ffbb-4414-80a1-4711a57d63a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.273915 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.273915 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-6dd139ce-2ca3-470e-b5db-f6d77d05e04c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.273915 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.274487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.274487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=15): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.274487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.277782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:32.277782 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:32.277782 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:32.277782 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:32.277782 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:32.277782 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:32.277782 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.277782 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:32.279103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.279103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:32.279103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:32.279103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:32.279103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.279103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:32.279103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:32.279103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.279103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:32.279103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:32.279103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:32.279103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.279734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:32.279734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:32.279734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:32.279734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:32.279734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:32.279734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:32.279734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:32.279734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:32.279734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:32.279734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:32.279734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:32.279734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:32.279734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:32.280047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.280047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:32.280047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.280047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:32.280047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:32.280047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:32.280047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:32.280047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:32.280047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.280047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:32.280047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:32.280047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.280047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:32.280047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:32.280047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:32.280442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:32.280950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:32.280950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:32.280950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:32.280950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:32.280950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.280950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:32.284756 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:32.284756 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:32.284756 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:32.284756 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:27:32.284756 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.284756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.288030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:32.288030 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:32.288030 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:32.288030 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:32.288030 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:32.288030 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:32.288030 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.288030 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:32.288874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.288874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:32.288874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:32.288874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:32.288874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.288874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:32.288874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:32.288874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:32.288874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.288874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:32.288874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:32.288874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:32.288874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:32.288874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:32.288874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:32.289995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:32.289995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:32.289995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:32.289995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:32.289995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:32.289995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:32.289995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:32.289995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.289995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:32.289995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.289995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:32.289995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:32.289995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:32.289995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.291178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:32.291178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:32.291178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:27:32.291178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:32.291178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:32.291178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.291178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:32.291178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:32.291178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:32.291178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.291178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:32.291178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:32.291178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:32.292157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:32.292157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:32.292157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:32.292157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:32.292157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:32.292157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:32.292157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:32.292157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:32.292157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:32.292157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:32.292157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.292157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:32.295434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.295434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:32.295434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:32.295434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:32.295434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:32.295434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:32.295434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.295434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:32.295434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:32.295434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.295434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:32.295434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:32.295434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.295434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:32.295434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:32.295434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:32.297093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:32.297598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:32.297598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:32.297598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:32.297598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:32.297598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:32.297598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.297598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:32.297598 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:32.297598 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:32.297598 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:32.297934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:32.297934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:32.297934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.297934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.297934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:32.297934 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:32.297934 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:32.297934 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:32.297934 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:32.297934 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:32.297934 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.297934 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:32.299171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.299171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:32.299171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:32.299171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:32.299171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.299171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:32.299171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:32.299171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:32.299171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.299171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:32.299171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:32.299171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:32.299171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:32.299171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:32.299171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:32.299593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:32.299593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:32.299593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:32.299593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:32.299593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:32.299593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:32.299593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:32.299593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.299593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:32.299593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.299593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:32.299593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:32.299593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:32.299593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.299991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:32.299991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:32.299991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:32.299991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:32.299991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.299991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:32.299991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:32.299991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:32.299991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.299991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:32.299991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:32.299991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:32.299991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:32.300316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:32.300316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:32.300316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:32.300316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:32.300316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:32.300316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:32.300316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:32.300316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:32.300316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:32.300316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.300316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.301652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:32.302881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:32.305876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:32.305876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:32.305876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.305876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:32.305876 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:32.305876 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:32.305876 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:32.305876 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.306423 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:32.306423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.306423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:32.306423 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:32.306423 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:32.306423 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:32.306423 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:32.306423 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:32.306423 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.306423 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:32.306423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.306423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:32.306423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:32.306423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:32.306881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.306881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:32.306881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:32.306881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.306881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:32.306881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:32.306881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:32.306881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.306881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:32.306881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:32.306881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:32.306881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:32.306881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:32.306881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:32.307224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:32.307224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:32.307224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:32.307224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:32.307224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:32.307224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:32.307224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:32.307224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.307224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:32.307520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.307520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:32.307520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:32.307520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:32.307520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:32.307520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:32.307520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.307520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:32.307520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:32.307520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.307520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:32.307520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:32.307520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.307520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:32.307520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:32.307520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:32.307989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:32.308466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:32.308466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:32.308466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:36.819744 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:36.823234 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:36.823389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:36.824288 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:36.903531 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:36.904594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:36.906414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:36.906414 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:36.906414 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:36.906414 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:36.906414 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:36.906414 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:36.906414 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:36.906414 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:36.907421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.083 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:36.907421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:36.907421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:36.907421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:36.907421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:36.907421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:36.907421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:36.907421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:36.907421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:36.907421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:36.907421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:36.907421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:36.907873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:36.907873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:36.907873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:36.907873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:36.907873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:36.907873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:36.907873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:36.907873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:36.907873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:36.907873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:36.907873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:36.907873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:36.907873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:36.908246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:36.908246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:36.908560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:36.908560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:36.908560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:36.908560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:36.908560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:36.908560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:36.908560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:36.908560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:36.908560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:36.908560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:36.908560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:36.908560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:36.908560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:36.908560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:36.908560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:36.908560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:36.909050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:36.909573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:36.909573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:36.909573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:37.269945 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:37.274824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:37.274824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:37.274824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.274824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.274824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.275079 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.275079 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.275079 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.275079 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-01b0936e-ffbb-4414-80a1-4711a57d63a5, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.275378 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.275378 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-6dd139ce-2ca3-470e-b5db-f6d77d05e04c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.275378 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.275693 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.276429 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.277453 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.277453 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-b19205b9-e0a3-4134-aa02-dd4aae79d2a4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.277453 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=14): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.277453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.279847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:37.279847 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:37.279847 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:37.279847 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:37.279847 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:37.279847 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:37.279847 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.279847 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:37.281027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.281027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:37.281027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:37.281027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:37.281027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.281027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:37.281027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:37.281027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.281027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:37.281027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:37.281027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:37.281027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.281486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:37.281486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:37.281486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:37.281486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:37.281486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:37.281486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:37.281486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:37.281486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:37.281486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:37.281486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:37.281486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:37.281486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:37.281486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:37.281842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.281842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:37.282528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.282528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:37.282528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:37.282528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:37.282528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:37.282528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:37.282528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.282528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:37.282528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:37.282528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.282528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:37.282528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:37.282528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.282528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:37.282528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:37.282528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:37.283958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:37.284438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:37.284438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.284438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:37.284438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:37.284438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:37.284438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:37.284438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:27:37.284438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.284677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.284941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:37.284941 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:37.284941 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:37.284941 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:37.284941 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:37.284941 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:37.284941 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.284941 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:37.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:37.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:37.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:37.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:37.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:37.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:37.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:37.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:37.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:37.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:37.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:37.285739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:37.286176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:37.286176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:37.286176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:37.286176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:37.286176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:37.286176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:37.286176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:37.286176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.286176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:37.286654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.286654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:37.286654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:37.286654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:37.286654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.286654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:37.286654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:37.286654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:27:37.286654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:37.286654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:37.286654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.286654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:37.287024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:37.287024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:37.287024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.287024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:37.287024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:37.287024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:37.287024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:37.287024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:37.287024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:37.287024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:37.287024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:37.287024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:37.287024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:37.287024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:37.287367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:37.287367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:37.287367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.287367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:37.287785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.287785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:37.287785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:37.287785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:37.287785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:37.287785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:37.287785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.287785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:37.287785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:37.287785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.287785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:37.287785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:37.287785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.287785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:37.287785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:37.287785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:37.288240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:37.288734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:37.288734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:37.288734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:37.288734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:37.288734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:37.288734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.288734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:37.296503 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:37.296908 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:37.297003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:37.297165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:37.299802 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.300117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.300944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:37.300944 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:37.300944 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:37.300944 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:37.300944 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:37.300944 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:37.300944 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.300944 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:37.302120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.302120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:37.302120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:37.302120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:37.302120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.302120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:37.302120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:37.302120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:37.302120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.302120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:37.302120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:37.302120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:37.302120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:37.302120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:37.302120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:37.303358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:37.303358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:37.303358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:37.303358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:37.303358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:37.303358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:37.303358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:37.303358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.303358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:37.306892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.306892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:37.306892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:37.306892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:37.306892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.306892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:37.306892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:37.306892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:37.306892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:37.306892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.306892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:37.310275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:37.310275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:37.310275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.310275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:37.310275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:37.310275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:37.310275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:37.310275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:37.310275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:37.310275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:37.310275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:37.310275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:37.310275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:37.310275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:37.310684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:37.310684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:37.310684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.310684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:37.310684 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:37.310684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.310684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:37.310684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:37.310684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:37.310684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:37.310684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:37.310684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.311239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:37.311819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:37.311819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:37.311819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:37.311819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:37.311819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:37.311819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:37.311819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:37.311819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:37.311819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:37.311819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:37.311819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:37.311819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:37.311819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.311819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:37.311819 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:37.312195 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:37.313130 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:37.314780 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.315432 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:37.315432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.318681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:37.318681 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:37.318681 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:37.318681 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:37.318681 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:37.318681 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:37.318681 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.318681 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:37.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:37.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:37.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:37.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:37.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:37.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:37.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:37.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:37.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.320368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:37.320368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:37.320368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:37.320368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:37.320368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:37.320368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:37.320368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:37.320368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:37.320368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:37.320368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:37.320368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:37.320368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:37.320368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:37.321140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.321140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:37.322325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.322325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:37.322325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:37.322325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:37.322325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:37.322325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:37.322325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.322325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:37.322325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:37.322325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.322325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:37.322325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:37.322325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.322325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:37.322325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:37.322325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:37.323349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:37.324184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:37.324184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:37.324184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:41.823322 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:41.823816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:41.823863 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:41.824660 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:41.890190 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:41.890190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:41.891455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:41.891455 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:41.891455 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:41.891455 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:41.891455 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:41.891455 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:41.891455 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:41.891455 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:41.891718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.067 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:41.891718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:41.891718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:41.891718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:41.891718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:41.891718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:41.891718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:41.891718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:41.891718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:41.891718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:41.891718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:41.891718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:41.891988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:41.891988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:41.891988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:41.891988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:41.891988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:41.891988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:41.891988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:41.891988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:41.891988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:41.891988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:41.891988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:41.891988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:41.891988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:41.892305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:41.892305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:41.894633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:41.894633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:41.894633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:41.894633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:41.894633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:41.894633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:41.894633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:41.894633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:41.894633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:41.894633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:41.894633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:41.894633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:41.894633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:41.894633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:41.894633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:41.894633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:41.895168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:41.895678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:41.895678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:41.895678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:42.270417 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:42.270885 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:42.271084 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:42.274877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.274877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.274877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.274877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.274877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.275155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.275155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.275155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-6dd139ce-2ca3-470e-b5db-f6d77d05e04c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.275155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.275155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.275346 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.275346 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.275346 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.275346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.275346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:42.275346 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:42.275346 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:42.275346 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:42.275346 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:42.275346 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:42.275346 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.275346 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:42.276072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.276072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:42.276072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:42.276072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:42.276072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.276072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:42.276072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:42.276072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.276072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:42.276072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:42.276072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:42.276072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.276436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:42.276436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:42.276436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:42.276436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:42.276436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:42.276436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:42.276436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:42.276436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:42.276436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:42.276436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:42.276436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:42.276436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:42.276436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:42.276794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.276794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:42.277256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.277256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:42.277256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:42.277256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:42.277256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:42.277256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:42.277256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.277256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:42.277256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:42.277256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.277256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:42.277256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:42.277256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.277256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:42.277256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:42.277256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:42.277711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:42.278173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:42.278173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.278173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:42.283728 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:42.284474 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:42.284748 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:42.284897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:27:42.286065 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.286065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.286261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:42.286261 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:42.286261 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:42.286261 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:42.286261 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:42.286261 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:42.286261 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.286261 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:42.287122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.287122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:42.287122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:42.287122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:42.287122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.287122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:42.287122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:42.287122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:42.287122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.287122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:42.287122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:42.287122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:42.287122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:42.287122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:42.287122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:42.287526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:42.287526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:42.287526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:42.287526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:42.287526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:42.287526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:42.287526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:42.287526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.287526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:42.287931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.287931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:42.287931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:42.287931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:42.287931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.287931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:42.287931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:42.287931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:27:42.287931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:42.287931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:42.287931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.287931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:42.288254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:42.288254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:42.288254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.288254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:42.288254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:42.288254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:42.288254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:42.288254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:42.288254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:42.288254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:42.288254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:42.288254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:42.288254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:42.288254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:42.288596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:42.288596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:42.288596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.288596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:42.289456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.289456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:42.289456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:42.289456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:42.289456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:42.289456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:42.289456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.289456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:42.289456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:42.289456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.289456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:42.289456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:42.289456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.289456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:42.289456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:42.289456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:42.290576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:42.291019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:42.291019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:42.291019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:42.291019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:42.291019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:42.291019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.291019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:42.296949 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:42.297371 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:42.297504 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:42.298637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:42.298637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.298637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.299405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:42.299405 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:42.299405 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:42.299405 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:42.299405 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:42.299405 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:42.299405 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.299405 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:42.299611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.299611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:42.299611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:42.299611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:42.299611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.299611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:42.299611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:42.299611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:42.299611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.299611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:42.299611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:42.299611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:42.299611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:42.299611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:42.299611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:42.299944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:42.299944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:42.299944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:42.299944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:42.299944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:42.299944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:42.299944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:42.299944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.299944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:42.300846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.300846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:42.300846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:42.300846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:42.300846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.300846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:42.300846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:42.300846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:42.300846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:42.300846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.300846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:42.301211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:42.301211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:42.301211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.301211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:42.301211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:42.301211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:42.301211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:42.301211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:42.301211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:42.301211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:42.301211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:42.301211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:42.301211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:42.301211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:42.301594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:42.301594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:42.301594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.301594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.302337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:42.303500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:42.304385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:42.304385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:42.304385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.304385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:42.309435 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:42.309697 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:42.310155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:42.310155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:42.310609 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.311007 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:42.311503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.312786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:42.312786 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:42.312786 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:42.312786 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:42.312786 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:42.312786 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:42.312786 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.312786 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:42.313601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.313601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:42.313601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:42.313601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:42.313601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.313601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:42.313601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:42.313601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.313601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:42.313601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:42.313601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:42.313601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.314066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:42.314066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:42.314066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:42.314066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:42.314066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:42.314066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:42.314066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:42.314066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:42.314066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:42.314066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:42.314066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:42.314066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:42.314066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:42.314374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.314374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:42.314703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.314703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:42.314703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:42.314703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:42.314703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:42.314703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:42.314703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.314703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:42.314703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:42.314703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.314703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:42.314703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:42.314703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.314703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:42.314703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:42.314703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:42.315141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:42.315612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:42.315612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:42.315612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:46.824236 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:46.825916 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:46.826341 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:46.828358 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:46.912491 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:46.912800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:46.914695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:46.914695 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:46.914695 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:46.914695 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:46.914695 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:46.914695 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:46.914695 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:46.914695 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:46.915810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.088 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:46.915810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:46.915810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:46.915810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:46.915810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:46.915810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:46.915810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:46.915810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:46.915810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:46.915810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:46.915810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:46.915810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:46.916150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:46.916150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:46.916150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:46.916150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:46.916150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:46.916150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:46.916150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:46.916150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:46.916150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:46.916150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:46.916150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:46.916150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:46.916150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:46.916458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:46.916458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:46.917342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:46.917342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:46.917342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:46.917342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:46.917342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:46.917342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:46.917342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:46.917342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:46.917342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:46.917342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:46.917342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:46.917342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:46.917342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:46.917342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:46.917342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:46.917342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:46.918460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:46.919486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:46.919486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:46.919486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:47.271292 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:47.271625 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:47.271719 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:47.273590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.273590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.273807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.273984 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.274055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.274147 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.274242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.274351 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-6dd139ce-2ca3-470e-b5db-f6d77d05e04c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.274508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.274628 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.274785 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.274896 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.275038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.275139 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.275354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.278623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:47.278623 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:47.278623 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:47.278623 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:47.278623 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:47.278623 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:47.278623 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.278623 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:47.278623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.278623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:47.278623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:47.278623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:47.278623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.278623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:47.278623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:47.279272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.279272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:47.279272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:47.279272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:47.279272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.279272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:47.279272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:47.279272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:47.279272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:47.279272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:47.279272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:47.279272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:47.279272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:47.279272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:47.279811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:47.279811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:47.279811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:47.279811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:47.279811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.279811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:47.279811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.279811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:47.279811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:47.279811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:47.279811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:47.279811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:47.279811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.279811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:47.280394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:47.281124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:47.281124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:47.281124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:47.281124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:47.281124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:47.281124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:47.281124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:47.281124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:47.281124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.281124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:47.285050 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:47.285622 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:47.285750 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:47.286065 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:27:47.286841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.287133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.287789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:47.287789 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:47.287789 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:47.287789 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:47.287789 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:47.287789 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:47.287789 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.287789 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:47.289404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.289404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:47.289404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:47.289404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:47.289404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.289404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:47.289404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:47.289404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:47.289404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.289404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:47.289404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:47.289404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:47.289404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:47.289404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:47.289404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:47.289884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:47.289884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:47.289884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:47.289884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:47.289884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:47.289884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:47.289884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:47.289884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.289884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:47.290239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.290239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:47.290239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:47.290239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:47.290239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.290239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:47.290239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:47.290239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:27:47.290239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:47.290239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:47.290239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.290239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:47.290999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:47.290999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:47.290999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.290999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:47.290999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:47.290999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:47.290999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:47.290999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:47.290999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:47.290999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:47.290999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:47.290999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:47.290999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:47.290999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:47.291357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:47.291357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:47.291357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.291357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:47.292171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.292171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:47.292171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:47.292171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:47.292171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:47.292171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:47.292171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.292171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:47.292171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:47.292171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.292171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:47.292171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:47.292171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.292171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:47.292171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:47.292171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:47.293031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:47.293643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:47.293643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:47.293643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:47.293643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:47.293643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:47.293643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.293643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:47.297270 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:47.297749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:47.297845 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:47.297985 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:47.298627 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.298984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.300109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:47.300109 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:47.300109 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:47.300109 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:47.300109 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:47.300109 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:47.300109 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.300109 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:47.300637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.300637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:47.300637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:47.300637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:47.300637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.300637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:47.300637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:47.300637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:47.300637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.300637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:47.300637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:47.300637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:47.300637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:47.300637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:47.300637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:47.301923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:47.301923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:47.301923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:47.301923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:47.301923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:47.301923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:47.301923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:47.301923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.301923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:47.301923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.301923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:47.301923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:47.301923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:47.301923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.303134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:47.303134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:47.303134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:47.303134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:47.303134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.303134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:47.303134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:47.303134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:47.303134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.303134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:47.303134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:47.303134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:47.303134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:47.303725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:47.303725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:47.303725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:47.303725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:47.303725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:47.303725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:47.303725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:47.303725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:47.303725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:47.303725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.303725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.304405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:47.304826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:47.305296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:47.305296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:47.305296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.305296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:47.311029 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:47.311885 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:47.312066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:47.312281 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:47.313638 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.313638 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:47.313638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.314307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:47.314307 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:47.314307 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:47.314307 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:47.314307 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:47.314307 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:47.314307 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.314307 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:47.315746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.315746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:47.315746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:47.315746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:47.315746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.315746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:47.315746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:47.315746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.315746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:47.315746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:47.315746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:47.315746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.316745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:47.316745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:47.316745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:47.316745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:47.316745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:47.316745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:47.316745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:47.316745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:47.316745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:47.316745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:47.316745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:47.316745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:47.316745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:47.317191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.317191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:47.317191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.317191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:47.317191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:47.317191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:47.317191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:47.317191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:47.317191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.317191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:47.317191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:47.317191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.317191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:47.317191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:47.317191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:47.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:47.318241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:47.318241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:47.318241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:47.318241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:47.318241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:47.318241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:51.825313 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:51.826742 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:51.826890 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:51.827714 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:51.968881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:51.969336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:51.975855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:51.975855 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:51.975855 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:51.975855 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:51.975855 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:51.975855 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:51.975855 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:51.975855 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:51.977141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.149 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:51.977141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:51.977141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:51.977141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:51.977141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:51.977141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:51.977141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:51.977141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:51.977141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:51.977141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:51.977141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:51.977141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:51.977458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:51.977458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:51.977458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:51.977458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:51.977458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:51.977458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:51.977458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:51.977458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:51.977458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:51.977458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:51.977458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:51.977458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:51.977458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:51.978659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:51.978659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:51.978659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:51.978659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:51.978659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:51.978659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:51.978659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:51.978659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:51.978659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:51.978659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:51.978659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:51.978659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:51.978659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:51.978659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:51.978659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:51.979150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:51.979713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:51.979713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:51.979713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:51.979713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:51.979713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:51.979713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:52.271781 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:52.271998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:52.272071 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:52.273426 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.273699 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.274130 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.274130 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.274269 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.274430 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.274759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.274911 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.275085 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.275256 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.275414 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.275587 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.275739 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.275895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.277530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:52.277530 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:52.277530 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:52.277530 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:52.277530 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:52.277530 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:52.277530 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.277530 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:52.278443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.278443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:52.278443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:52.278443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:52.278443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.278443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:52.278443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:52.278443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.278443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:52.278443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:52.278443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:52.278443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.278918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:52.278918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:52.278918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:52.278918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:52.278918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:52.278918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:52.278918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:52.278918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:52.278918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:52.278918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:52.278918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:52.278918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:52.278918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:52.279306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.279306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:52.279306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.279306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:52.279306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:52.279306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:52.279306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:52.279306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:52.279306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.279306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:52.279306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:52.279306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.279306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:52.279306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:52.279306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:52.279813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:52.280276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:52.280276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:52.280276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:52.280276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:52.280276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.280276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:52.286062 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:52.286062 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:52.286062 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:52.286326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:27:52.286982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.286982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.287605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:52.287605 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:52.287605 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:52.287605 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:52.287605 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:52.287605 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:52.287605 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.287605 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:52.288462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.288462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:52.288462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:52.288462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:52.288462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.288462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:52.288462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:52.288462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:52.288462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.288462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:52.288462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:52.288462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:52.288462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:52.288462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:52.288462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:52.289059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:52.289059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:52.289059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:52.289059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:52.289059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:52.289059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:52.289059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:52.289059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.289059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:52.289671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.289671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:52.289671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:52.289671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:52.289671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.289671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:52.289671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:52.289671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:27:52.289671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:52.289671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:52.289671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.289671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:52.290084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:52.290084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:52.290084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.290084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:52.290084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:52.290084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:52.290084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:52.290084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:52.290084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:52.290084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:52.290084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:52.290084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:52.290084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:52.290084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:52.290496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:52.290496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:52.290496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.290496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:52.290657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.290657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:52.290657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:52.290657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:52.290657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:52.290657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:52.290657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.290657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:52.290657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:52.290657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.290657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:52.290657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:52.290657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.290657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:52.290657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:52.290657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:52.291095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:52.292221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:52.292221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:52.292221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:52.292221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:52.292221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:52.292221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.292221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:52.297930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:52.298890 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:52.299174 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:52.299257 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:52.299626 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.299840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.300394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:52.300394 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:52.300394 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:52.300394 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:52.300394 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:52.300394 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:52.300394 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.300394 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:52.301156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.301156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:52.301156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:52.301156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:52.301156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.301156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:52.301156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:52.301156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:52.301156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.301156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:52.301156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:52.301156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:52.301156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:52.301156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:52.301156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:52.301806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:52.301806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:52.301806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:52.301806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:52.301806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:52.301806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:52.301806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:52.301806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.301806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:52.302131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.302131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:52.302131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:52.302131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:52.302131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.302131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:52.302131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:52.302131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:52.302131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:52.302131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.302131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:52.302429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:52.302429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:52.302429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.302429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:52.302429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:52.302429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:52.302429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:52.302429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:52.302429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:52.302429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:52.302429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:52.302429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:52.302429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:52.302429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:52.302811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:52.302811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:52.302811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.302811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:52.302811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.302811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:52.302811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:52.302811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:52.302811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:52.302811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:52.302811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.302811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:52.302811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:52.302811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:52.303220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:52.304087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:52.304087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:52.304087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:52.304087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:52.304087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:52.304087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:52.304087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:52.304087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:52.304087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:52.304087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.304087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:52.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:52.312905 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:52.312905 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:52.313181 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:52.313805 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.313805 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:52.314322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:52.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:52.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:52.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:52.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:52.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:52.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:52.315826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.315826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:52.315826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:52.315826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:52.315826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.315826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:52.315826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:52.315826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.315826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:52.315826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:52.315826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:52.315826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.316270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:52.316270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:52.316270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:52.316270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:52.316270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:52.316270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:52.316270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:52.316270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:52.316270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:52.316270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:52.316270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:52.316270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:52.316270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:52.316677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.316677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:52.316895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.316895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:52.316895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:52.316895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:52.316895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:52.316895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:52.316895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.316895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:52.316895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:52.316895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.316895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:52.316895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:52.316895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.316895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:52.316895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:52.316895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:52.317452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:52.318373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:52.318373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:52.318373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:56.827562 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:56.827562 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:56.827562 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:56.829126 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:56.946066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:56.946806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:56.947060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:56.947060 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:56.947060 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:56.947060 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:56.947060 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:56.947060 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:56.947060 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:56.947060 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:56.948575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.120 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:56.948575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:56.948575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:56.948575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:56.948575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:56.948575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:56.948575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:56.948575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:56.948575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:56.948575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:56.948575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:56.948575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:56.949051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:56.949051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:56.949051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:56.949051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:56.949051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:56.949051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:56.949051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:56.949051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:56.949051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:56.949051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:56.949051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:56.949051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:56.949051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:56.949617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:56.949617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:56.949937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:56.949937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:56.949937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:56.949937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:56.949937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:56.949937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:56.949937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:56.949937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:56.949937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:56.949937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:56.949937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:56.949937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:56.949937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:56.949937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:56.949937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:56.949937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:56.950465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:56.952180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:56.952180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:56.952180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:57.272284 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:57.274420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:57.274420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:57.274859 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.275139 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.275712 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.275712 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.275712 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.275712 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.276153 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.276153 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.276153 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.276284 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.276337 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.276458 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.276595 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.276697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.279661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:57.279661 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:57.279661 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:57.279661 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:57.279661 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:57.279661 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:57.279661 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.279661 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:57.279661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.279661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:57.279661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:57.279661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:57.279661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.279661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:57.279661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:57.280585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.280585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:57.280585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:57.280585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:57.280585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.280585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:57.280585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:57.280585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:57.280585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:57.280585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:57.280585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:57.280585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:57.280585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:57.280585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:57.280916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:57.280916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:57.280916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:57.280916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:57.280916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.280916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:57.280916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.280916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:57.280916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:57.280916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:57.280916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:57.280916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:57.280916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.280916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:57.281316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:57.281821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:57.281821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:57.281821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:57.281821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:57.281821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:57.281821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:57.281821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:57.281821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:57.281821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.281821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:57.286167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:57.286782 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:57.288114 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:57.288482 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:27:57.295253 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.295475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.296335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:57.296335 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:57.296335 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:57.296335 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:57.296335 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:57.296335 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:57.296335 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.296335 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:57.297891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.297891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:57.297891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:57.297891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:57.297891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.297891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:57.297891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:57.297891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:57.297891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.297891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:57.297891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:57.297891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:57.297891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:57.297891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:57.297891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:57.298327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:57.298327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:57.298327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:57.298327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:57.298327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:57.298327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:57.298327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:57.298327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.298327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:57.299014 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:57.301434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.301434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:57.301434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:57.301434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:57.301434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.301434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:57.301434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:57.301434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:27:57.301434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:57.301434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:57.301434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.301434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:57.301922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:57.301922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:57.301922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.301922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:57.301922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:57.301922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:57.301922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:57.301922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:57.301922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:57.301922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:57.301922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:57.301922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:57.301922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:57.301922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:57.302378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:57.302378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:57.302378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.302378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:57.302378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.302378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:57.302378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:57.302378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:57.302378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:57.302378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:57.302378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.302378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:57.302378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:57.302378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:57.302833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:57.303406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:57.303406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:57.303406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:57.303406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:57.303406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:57.303406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:57.303406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:57.303406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:57.303406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:57.303406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:57.303406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.303406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:57.303406 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:57.303406 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:57.303881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:57.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.304783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.305809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:57.305809 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:57.305809 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:57.305809 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:57.305809 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:57.305809 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:57.305809 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.305809 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:57.308606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.308606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:27:57.308606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:57.308606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:57.308606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.308606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:57.308606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:57.308606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:27:57.308606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.308606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:27:57.308606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:57.308606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:27:57.308606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:57.308606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:27:57.308606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:27:57.309106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:57.309106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:27:57.309106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:57.309106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:27:57.309106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:27:57.309106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:57.309106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:27:57.309106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.309106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:27:57.311707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.311707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:57.311707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:57.311707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:57.311707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.311707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:57.311707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:57.311707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:57.311707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:57.311707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.311707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:57.312066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:57.312066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:57.312066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.312066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:57.312066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:57.312066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:57.312066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:57.312066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:57.312066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:57.312066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:57.312066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:57.312066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:57.312066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:57.312066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:57.312417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:57.312417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:57.312417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.312417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.315946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:57.316475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:57.317262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:57.317262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:57.317262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.317262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:27:57.317262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:27:57.318807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:27:57.318807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:27:57.318807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:27:57.319321 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.319618 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:27:57.320017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.321216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:27:57.321216 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:57.321216 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:27:57.321216 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:27:57.321216 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:57.321216 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:27:57.321216 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.321216 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:27:57.322775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.322775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:27:57.322775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:57.322775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:27:57.322775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.322775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:57.322775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:57.322775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.322775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:57.322775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:57.322775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:27:57.322775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.323053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:27:57.323053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:57.323053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:27:57.323053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:57.323053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:27:57.323053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:27:57.323053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:57.323053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:27:57.323053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:57.323053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:27:57.323053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:27:57.323053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:57.323053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:27:57.324028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.324028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:27:57.324700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.324700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:27:57.324700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:27:57.324700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:27:57.324700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:27:57.324700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:27:57.324700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.324700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:27:57.324700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:27:57.324700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.324700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:27:57.324700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:27:57.324700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.324700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:27:57.324700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:27:57.324700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:27:57.325030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:27:57.325460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:27:57.325460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:27:57.325460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:01.827586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:01.828109 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:01.828763 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:01.829576 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:01.914086 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:01.914460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:01.915672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:01.915672 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:01.915672 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:01.915672 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:01.915672 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:01.915672 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:01.915672 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:01.915672 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:01.916974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.087 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:01.916974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:01.916974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:01.916974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:01.916974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:01.916974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:01.916974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:01.916974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:01.916974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:01.916974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:01.916974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:01.916974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:01.919213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:01.919213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:01.919213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:01.919213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:01.919213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:01.919213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:01.919213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:01.919213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:01.919213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:01.919213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:01.919213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:01.919213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:01.919213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:01.919792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:01.919792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:01.919792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:01.919792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:01.919792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:01.919792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:01.919792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:01.919792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:01.919792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:01.919792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:01.919792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:01.919792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:01.919792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:01.919792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:01.919792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:01.920415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:01.921395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:01.921395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:01.921395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:01.921395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:01.921395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:01.921395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:02.274929 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:02.275482 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:02.277235 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:02.277769 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.278017 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.278094 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.278226 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.278329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.281170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.281170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.281170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.281170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.281170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.281431 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.281431 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.281431 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.281431 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.281431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.281431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:02.281431 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:02.281431 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:02.281431 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:02.281691 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:02.281691 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:02.281691 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.281691 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:02.281913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.281913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:02.281913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:02.281913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:02.281913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.281913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:02.281913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:02.281913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.281913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:02.281913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:02.281913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:02.281913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.282203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:02.282203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:02.282203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:02.282203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:02.282203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:02.282203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:02.282203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:02.282203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:02.282203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:02.282203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:02.282203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:02.282203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:02.282203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:02.282505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.282505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:02.283272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.283272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:02.283272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:02.283272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:02.283272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:02.283272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:02.283272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.283272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:02.283272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:02.283272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.283272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:02.283272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:02.283272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.283272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:02.283272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:02.283272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:02.283619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:02.283973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:02.283973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.283973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:02.287810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:02.287810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:02.287810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:02.287810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:28:02.288140 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.288415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.288941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:02.288941 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:02.288941 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:02.288941 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:02.288941 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:02.288941 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:02.288941 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.288941 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:02.289795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.289795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:02.289795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:02.289795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:02.289795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.289795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:02.289795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:02.289795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:02.289795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.289795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:02.289795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:02.289795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:02.289795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:02.289795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:02.289795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:02.290396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:02.290396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:02.290396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:02.290396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:02.290396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:02.290396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:02.290396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:02.290396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.290396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:02.290979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.290979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:02.290979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:02.290979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:02.290979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.290979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:02.290979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:02.290979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:28:02.290979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:02.290979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:02.290979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.290979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:02.291316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:02.291316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:02.291316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.291316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:02.291316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:02.291316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:02.291316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:02.291316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:02.291316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:02.291316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:02.291316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:02.291316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:02.291316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:02.291316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:02.291576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:02.291576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:02.291576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.291576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:02.292302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.292302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:02.292302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:02.292302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:02.292302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:02.292302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:02.292302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.292302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:02.292302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:02.292302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.292302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:02.292302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:02.292302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.292302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:02.292302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:02.292302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:02.292657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:02.293098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:02.293098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:02.293098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:02.293098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:02.293098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:02.293098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.293098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:02.301319 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:02.301782 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:02.302005 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:02.302156 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:02.302721 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.303097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.303697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:02.303697 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:02.303697 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:02.303697 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:02.303697 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:02.303697 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:02.303697 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.303697 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:02.304482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.304482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:02.304482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:02.304482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:02.304482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.304482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:02.304482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:02.304482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:02.304482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.304482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:02.304482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:02.304482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:02.304482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:02.304482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:02.304482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:02.305016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:02.305016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:02.305016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:02.305016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:02.305016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:02.305016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:02.305016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:02.305016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.305016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:02.305347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.305347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:02.305347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:02.305347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:02.305347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.305347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:02.305347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:02.305347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:02.305347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:02.305347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.305347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:02.305777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:02.305777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:02.305777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.305777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:02.305777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:02.305777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:02.305777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:02.305777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:02.305777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:02.305777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:02.305777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:02.305777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:02.305777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:02.305777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:02.306233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:02.306233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:02.306233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.306233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.306397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:02.307026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:02.308134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:02.308134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:02.308134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.308134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:02.318930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:02.318930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:02.318930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:02.319148 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:02.321991 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.322326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:02.324259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.325282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:02.325282 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:02.325282 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:02.325282 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:02.325282 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:02.325282 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:02.325282 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.325282 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:02.326597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.326597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:02.326597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:02.326597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:02.326597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.326597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:02.326597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:02.326597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.326597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:02.326597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:02.326597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:02.326597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.327107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:02.327107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:02.327107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:02.327107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:02.327107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:02.327107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:02.327107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:02.327107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:02.327107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:02.327107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:02.327107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:02.327107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:02.327107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:02.329448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.329448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:02.329448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.329448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:02.329448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:02.329448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:02.329448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:02.329448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:02.329448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.329448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:02.329448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:02.329448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.329448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:02.329448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:02.329448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:02.330061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:02.331951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:02.331951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:02.331951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:02.331951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:02.331951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:02.331951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:06.829608 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:06.831011 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:06.831011 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:06.831690 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:06.924283 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:06.924283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:06.926184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:06.926184 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:06.926184 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:06.926184 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:06.926184 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:06.926184 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:06.926184 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:06.926184 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:06.927215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.096 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:06.927215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:06.927215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:06.927215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:06.927215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:06.927215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:06.927215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:06.927215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:06.927215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:06.927215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:06.927215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:06.927215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:06.927532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:06.927532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:06.927532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:06.927532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:06.927532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:06.927532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:06.927532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:06.927532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:06.927532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:06.927532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:06.927532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:06.927532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:06.927532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:06.927831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:06.927831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:06.929519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:06.929519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:06.929519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:06.929519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:06.929519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:06.929519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:06.929519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:06.929519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:06.929519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:06.929519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:06.929519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:06.929519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:06.929519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:06.929519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:06.929519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:06.929519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:06.929905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:06.930370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:06.930370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:06.930370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:07.275499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:07.275689 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:07.275750 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:07.281376 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.281376 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.282044 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.282044 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.282044 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.283676 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.283676 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.283676 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.285995 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.286288 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.287066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.287391 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.288118 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.288390 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.289425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.290489 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:07.291435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:07.291435 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:07.291435 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:07.291435 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:07.291435 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:07.291435 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:07.291435 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.291435 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:07.292290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.015 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.292290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:07.292290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:07.292290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:07.292290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.292290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:07.292290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:07.292290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.292290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:07.292290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:07.292290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:07.292290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.293711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:07.293711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:07.293711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:07.293711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:07.293711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:07.293711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:07.293711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:07.293711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:07.293711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:07.293711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:07.293711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:07.293711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:07.293711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:07.294178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.294178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:07.294178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.294178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:07.294178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:07.294178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:07.294178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:07.294178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:07.294178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.294178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:07.294178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:07.294178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.294178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:07.294178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:07.294178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:07.294699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:07.295348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:07.295348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:07.295348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:07.295348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:07.295348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.295348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:07.295348 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:07.295348 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:07.295348 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:28:07.296816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.297105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.299895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:07.299895 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:07.299895 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:07.299895 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:07.299895 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:07.299895 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:07.299895 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.299895 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:07.301168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.301168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:07.301168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:07.301168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:07.301168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.301168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:07.301168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:07.301168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:07.301168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.301168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:07.301168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:07.301168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:07.301168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:07.301168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:07.301168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:07.301749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:07.301749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:07.301749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:07.301749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:07.301749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:07.301749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:07.301749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:07.301749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.301749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:07.302467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.302467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:07.302467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:07.302467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:07.302467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.302467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:07.302467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:07.302467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:28:07.302467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:07.302467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:07.302467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.302467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:07.303230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:07.303230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:07.303230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.303230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:07.303230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:07.303230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:07.303230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:07.303230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:07.303230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:07.303230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:07.303230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:07.303230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:07.303230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:07.303230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:07.305167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:07.305167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:07.305167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.305167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:07.305167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.305167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:07.305167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:07.305167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:07.305167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:07.305167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:07.305167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.305167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:07.305167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:07.305167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:07.305461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:07.306886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:07.306886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:07.306886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:07.306886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:07.306886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:07.306886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:07.306886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:07.306886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:07.306886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:07.306886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:07.306886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.306886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:07.306886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:07.306886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:07.307327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:07.307327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:07.307327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.307327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.310092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:07.310092 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:07.310092 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:07.310092 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:07.310092 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:07.310092 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:07.310092 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.310092 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:07.311000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.311000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:07.311000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:07.311000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:07.311000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.311000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:07.311000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:07.311000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:07.311000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.311000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:07.311000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:07.311000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:07.311000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:07.311000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:07.311000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:07.311638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:07.311638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:07.311638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:07.311638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:07.311638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:07.311638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:07.311638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:07.311638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.311638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:07.314216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.314216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:07.314216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:07.314216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:07.314216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.314216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:07.314216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:07.314216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:07.314216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:07.314216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.314216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:07.315623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:07.315623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:07.315623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.315623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:07.315623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:07.315623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:07.315623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:07.315623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:07.315623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:07.315623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:07.315623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:07.315623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:07.315623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:07.315623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:07.316050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:07.316050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:07.316050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.316050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.319307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:07.320183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:07.320807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:07.320807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:07.320807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.320807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:07.320807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:07.321233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:07.321423 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:07.321879 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:07.323397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.324296 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:07.325584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.327136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:07.327136 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:07.327136 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:07.327136 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:07.327136 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:07.327136 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:07.327136 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.327136 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:07.328865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.328865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:07.328865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:07.328865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:07.328865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.328865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:07.328865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:07.328865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.328865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:07.328865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:07.328865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:07.328865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.329413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:07.329413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:07.329413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:07.329413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:07.329413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:07.329413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:07.329413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:07.329413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:07.329413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:07.329413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:07.329413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:07.329413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:07.329413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:07.329751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.329751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:07.329751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.329751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:07.329751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:07.329751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:07.329751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:07.329751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:07.329751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.329751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:07.329751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:07.329751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.329751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:07.329751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:07.329751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:07.331818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:07.332804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:07.332804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:07.332804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:07.332804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:07.332804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:07.332804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:11.830257 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:11.830918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:11.830918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:11.831611 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:11.897512 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:11.897821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:11.899057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:11.899057 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:11.899057 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:11.899057 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:11.899057 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:11.899057 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:11.899057 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:11.899057 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:11.899667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.068 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:11.899667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:11.899667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:11.899667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:11.899667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:11.899667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:11.899667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:11.899667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:11.899667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:11.899667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:11.899667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:11.899667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:11.900054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:11.900054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:11.900054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:11.900054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:11.900054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:11.900054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:11.900054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:11.900054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:11.900054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:11.900054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:11.900054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:11.900054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:11.900054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:11.900408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:11.900408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:11.901195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:11.901195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:11.901195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:11.901195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:11.901195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:11.901195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:11.901195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:11.901195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:11.901195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:11.901195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:11.901195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:11.901195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:11.901195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:11.901195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:11.901195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:11.901195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:11.901747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:11.902465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:11.902465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:11.902465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:12.276476 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:12.277345 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:12.277527 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:12.282765 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.283368 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.283640 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.283832 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.286645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.286645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.286645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.286645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.286645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.286952 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.286952 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.286952 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.286952 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.286952 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.287149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.294392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:12.294392 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:12.294392 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:12.294392 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:12.294392 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:12.294392 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:12.294392 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.294392 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:12.295289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.016 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.295289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:12.295289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:12.295289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:12.295289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.295289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:12.295289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:12.295289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.295289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:12.295289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:12.295289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:12.295289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.295848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:12.295848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:12.295848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:12.295848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:12.295848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:12.295848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:12.295848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:12.295848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:12.295848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:12.295848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:12.295848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:12.295848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:12.295848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:12.296265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.296265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:12.296564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.296564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:12.296564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:12.296564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:12.296564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:12.296564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:12.296564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.296564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:12.296564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:12.296564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.296564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:12.296564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:12.296564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.296564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:12.296564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:12.296564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:12.297032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:12.298065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:12.298065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.298065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:12.298065 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:12.299210 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:12.299210 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:12.299210 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:28:12.299565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.299792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.300398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:12.300398 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:12.300398 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:12.300398 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:12.300398 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:12.300398 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:12.300398 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.300398 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:12.301429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.301429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:12.301429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:12.301429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:12.301429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.301429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:12.301429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:12.301429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:12.301429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.301429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:12.301429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:12.301429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:12.301429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:12.301429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:12.301429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:12.302362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:12.302362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:12.302362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:12.302362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:12.302362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:12.302362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:12.302362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:12.302362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.302362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:12.302362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.302362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:12.302362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:12.302362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:12.302362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.302766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:12.302766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:12.302766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:28:12.302766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:12.302766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:12.302766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.302766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:12.302766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:12.302766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:12.302766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.302766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:12.302766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:12.302766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:12.303114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:12.303114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:12.303114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:12.303114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:12.303114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:12.303114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:12.303114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:12.303114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:12.303114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:12.303114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:12.303114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.303114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:12.303565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.303565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:12.303565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:12.303565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:12.303565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:12.303565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:12.303565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.303565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:12.303565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:12.303565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.303565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:12.303565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:12.303565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.303565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:12.303565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:12.303565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:12.304480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:12.305468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:12.305468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:12.305468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:12.305468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:12.305468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:12.305468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.305468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:12.305468 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:12.306038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:12.306600 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:12.306600 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:12.306822 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.307143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.307720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:12.307720 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:12.307720 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:12.307720 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:12.307720 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:12.307720 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:12.307720 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.307720 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:12.308719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.308719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:12.308719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:12.308719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:12.308719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.308719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:12.308719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:12.308719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:12.308719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.308719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:12.308719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:12.308719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:12.308719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:12.308719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:12.308719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:12.309157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:12.309157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:12.309157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:12.309157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:12.309157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:12.309157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:12.309157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:12.309157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.309157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:12.310026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.310026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:12.310026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:12.310026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:12.310026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.310026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:12.310026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:12.310026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:12.310026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:12.310026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.310026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:12.310384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:12.310384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:12.310384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.310384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:12.310384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:12.310384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:12.310384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:12.310384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:12.310384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:12.310384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:12.310384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:12.310384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:12.310384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:12.310384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:12.310731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:12.310731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:12.310731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.310731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.311592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:12.312294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:12.313389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:12.313389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:12.313389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.313389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:12.321345 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:12.321883 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:12.321883 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:12.322149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:12.323691 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.323691 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:12.323691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.324383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:12.324383 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:12.324383 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:12.324383 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:12.324383 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:12.324383 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:12.324383 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.324383 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:12.326624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.326624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:12.326624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:12.326624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:12.326624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.326624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:12.326624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:12.326624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.326624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:12.326624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:12.326624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:12.326624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.327130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:12.327130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:12.327130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:12.327130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:12.327130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:12.327130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:12.327130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:12.327130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:12.327130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:12.327130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:12.327130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:12.327130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:12.327130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:12.327593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.327593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:12.327593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.327593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:12.327593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:12.327593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:12.327593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:12.327593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:12.327593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.327593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:12.327593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:12.327593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.327593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:12.327593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:12.327593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:12.328204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:12.328846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:12.328846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:12.328846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:12.328846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:12.328846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:12.328846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:16.831495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:16.832426 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:16.832581 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:16.833568 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:16.902658 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:16.903050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:16.904862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:16.904862 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:16.904862 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:16.904862 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:16.904862 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:16.904862 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:16.904862 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:16.904862 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:16.905478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.071 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:16.905478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:16.905478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:16.905478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:16.905478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:16.905478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:16.905478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:16.905478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:16.905478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:16.905478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:16.905478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:16.905478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:16.905781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:16.905781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:16.905781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:16.905781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:16.905781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:16.905781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:16.905781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:16.905781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:16.905781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:16.905781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:16.905781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:16.905781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:16.905781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:16.906591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:16.906591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:16.906591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:16.906591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:16.906591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:16.906591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:16.906591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:16.906591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:16.906591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:16.906591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:16.906591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:16.906591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:16.906591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:16.906591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:16.906591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:16.908052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:16.909003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:16.909003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:16.909003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:16.909003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:16.909003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:16.909003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:17.277876 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:17.279179 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:17.279354 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:17.285880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.285880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.287918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.287918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.287918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.287918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.287918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.288169 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.288169 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.288169 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.288169 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.288169 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.288357 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.288357 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.288357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.289058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:17.289058 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:17.289058 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:17.289058 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:17.289058 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:17.289058 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:17.289058 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.289058 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:17.290306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.290306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:17.290306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:17.290306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:17.290306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.290306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:17.290306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:17.290306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.290306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:17.290306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:17.290306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:17.290306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.290602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:17.290602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:17.290602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:17.290602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:17.290602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:17.290602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:17.290602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:17.290602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:17.290602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:17.290602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:17.290602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:17.290602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:17.290602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:17.290816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.290816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:17.292953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.292953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:17.292953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:17.292953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:17.292953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:17.292953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:17.292953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.292953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:17.292953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:17.292953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.292953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:17.292953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:17.292953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.292953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:17.292953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:17.292953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:17.293322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:17.293737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:17.293737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.293737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:17.301946 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:17.302394 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:17.302535 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:17.302840 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:28:17.303483 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.303836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.304568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:17.304568 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:17.304568 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:17.304568 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:17.304568 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:17.304568 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:17.304568 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.304568 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:17.305359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.305359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:17.305359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:17.305359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:17.305359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.305359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:17.305359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:17.305359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:17.305359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.305359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:17.305359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:17.305359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:17.305359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:17.305359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:17.305359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:17.305836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:17.305836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:17.305836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:17.305836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:17.305836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:17.305836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:17.305836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:17.305836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.305836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:17.307002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.307002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:17.307002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:17.307002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:17.307002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.307002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:17.307002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:17.307002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:28:17.307002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:17.307002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:17.307002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.307002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:17.307467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:17.307467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:17.307467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.307467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:17.307467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:17.307467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:17.307467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:17.307467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:17.307467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:17.307467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:17.307467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:17.307467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:17.307467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:17.307467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:17.307880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:17.307880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:17.307880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.307880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:17.311348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.311348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:17.311348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:17.311348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:17.311348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:17.311348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:17.311348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.311348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:17.311348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:17.311348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.311348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:17.311348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:17.311348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.311348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:17.311348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:17.311348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:17.312343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:17.312958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:17.312958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:17.312958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:17.312958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:17.312958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:17.312958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.312958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:17.312958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:17.312958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:17.312958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:17.313331 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:17.313331 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.313331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.313331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:17.313331 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:17.313331 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:17.313331 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:17.313331 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:17.313331 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:17.313331 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.313331 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:17.313331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:17.313709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:17.314298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:17.314298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:17.314298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:17.314298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:17.314298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.314298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:17.314298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.314298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:17.314298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:17.314298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:17.314298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.314298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:17.314298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:17.314771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:17.314771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:17.314771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.314771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:17.314771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:17.314771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:17.314771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.314771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:17.314771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:17.314771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:17.314771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:17.314771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:17.314771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:17.315168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:17.315168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:17.315168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:17.315168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:17.315168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:17.315168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:17.315168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:17.315168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.315168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.316294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:17.317216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:17.317728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:17.317728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:17.317728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.317728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:17.321824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:17.322638 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:17.322818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:17.323106 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:17.326279 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.326279 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:17.326978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.328161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:17.328161 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:17.328161 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:17.328161 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:17.328161 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:17.328161 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:17.328161 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.328161 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:17.329472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.329472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:17.329472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:17.329472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:17.329472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.329472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:17.329472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:17.329472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.329472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:17.329472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:17.329472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:17.329472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.329895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:17.329895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:17.329895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:17.329895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:17.329895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:17.329895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:17.329895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:17.329895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:17.329895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:17.329895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:17.329895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:17.329895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:17.329895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:17.330231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.330231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:17.330661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.330661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:17.330661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:17.330661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:17.330661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:17.330661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:17.330661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.330661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:17.330661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:17.330661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.330661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:17.330661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:17.330661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.330661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:17.330661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:17.330661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:17.331177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:17.332010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:17.332010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:17.332010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:21.832856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:21.835707 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:21.835707 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:21.835707 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:21.836905 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=240 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:28:21.914020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:21.914341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:21.915580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:21.915580 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:21.915580 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:21.915580 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:21.915580 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:21.915580 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:21.915580 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:21.915580 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:21.916181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.081 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:21.916181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:21.916181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:21.916181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:21.916181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:21.916181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:21.916181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:21.916181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:21.916181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:21.916181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:21.916181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:21.916181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:21.916429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:21.916429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:21.916429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:21.916429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:21.916429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:21.916429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:21.916429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:21.916429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:21.916429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:21.916429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:21.916429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:21.916429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:21.916429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:21.916661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:21.916661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:21.917287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:21.917287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:21.917287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:21.917287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:21.917287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:21.917287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:21.917287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:21.917287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:21.917287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:21.917287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:21.917287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:21.917287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:21.917287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:21.917287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:21.917287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:21.917287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:21.917634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:21.917998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:21.917998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:21.917998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:22.279238 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:22.279499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:22.279674 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:22.282745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.283041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.283251 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.283436 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.283712 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.283869 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.284019 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.284160 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.284301 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.284460 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.284742 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.284945 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.285090 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.285311 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=13): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.285451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.287230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:22.287230 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:22.287230 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:22.287230 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:22.287230 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:22.287230 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:22.287230 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.287230 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:22.288405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.288405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:22.288405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:22.288405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:22.288405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.288405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:22.288405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:22.288405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.288405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:22.288405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:22.288405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:22.288405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.288786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:22.288786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:22.288786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:22.288786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:22.288786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:22.288786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:22.288786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:22.288786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:22.288786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:22.288786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:22.288786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:22.288786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:22.288786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:22.289206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.289206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:22.289362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.289362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:22.289362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:22.289362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:22.289362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:22.289362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:22.289362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.289362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:22.289362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:22.289362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.289362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:22.289362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:22.289362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.289362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:22.289362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:22.289362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:22.289894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:22.290388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:22.290388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.290388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:22.302569 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:22.304886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:22.305027 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:22.305158 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:28:22.305550 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.305761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.306746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:22.306746 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:22.306746 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:22.306746 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:22.306746 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:22.306746 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:22.306746 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.306746 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:22.311644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.311644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:22.311644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:22.311644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:22.311644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.311644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:22.311644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:22.311644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:22.311644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.311644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:22.311644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:22.311644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:22.311644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:22.311644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:22.311644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:22.312230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:22.312230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:22.312230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:22.312230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:22.312230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:22.312230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:22.312230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:22.312230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.312230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:22.312230 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:22.312890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.312890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:22.312890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:22.312890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:22.312890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.312890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:22.312890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:22.312890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:28:22.312890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:22.312890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:22.312890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.312890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:22.313378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:22.313378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:22.313378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.313378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:22.313378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:22.313378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:22.313378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:22.313378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:22.313378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:22.313378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:22.313378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:22.313378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:22.313378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:22.313378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:22.313744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:22.313744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:22.313744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.313744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:22.314057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.314057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:22.314057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:22.314057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:22.314057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:22.314057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:22.314057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.314057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:22.314057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:22.314057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.314057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:22.314057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:22.314057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.314057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:22.314057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:22.314057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:22.314521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:22.315200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:22.315200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:22.315200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:22.315200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:22.315200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:22.315200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.315200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:22.315200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:22.315200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:22.315200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:22.315490 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.315490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.315802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:22.315802 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:22.315802 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:22.315802 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:22.315802 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:22.315802 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:22.315802 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.315802 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:22.316673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.316673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:22.316673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:22.316673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:22.316673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.316673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:22.316673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:22.316673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:22.316673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.316673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:22.316673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:22.316673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:22.316673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:22.316673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:22.316673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:22.317298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:22.317298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:22.317298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:22.317298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:22.317298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:22.317298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:22.317298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:22.317298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.317298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:22.317641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.317641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:22.317641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:22.317641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:22.317641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.317641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:22.317641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:22.317641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:22.317641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:22.317641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.317641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:22.318262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:22.318262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:22.318262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.318262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:22.318262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:22.318262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:22.318262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:22.318262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:22.318262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:22.318262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:22.318262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:22.318262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:22.318262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:22.318262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:22.318794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:22.318794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:22.318794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.318794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:22.318794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.318794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:22.318794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:22.318794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:22.318794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:22.318794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:22.318794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.318794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:22.318794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:22.318794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:22.319294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:22.319916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:22.319916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:22.319916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:22.319916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:22.319916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:22.319916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:22.319916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:22.319916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:22.319916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:22.319916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.319916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:22.325374 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:22.325706 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:22.325865 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:22.326076 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:22.326414 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.326705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:22.327195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.327702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:22.327702 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:22.327702 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:22.327702 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:22.327702 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:22.327702 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:22.327702 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.327702 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:22.328660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.328660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:22.328660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:22.328660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:22.328660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.328660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:22.328660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:22.328660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.328660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:22.328660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:22.328660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:22.328660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.328938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:22.328938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:22.328938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:22.328938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:22.328938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:22.328938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:22.328938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:22.328938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:22.328938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:22.328938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:22.328938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:22.328938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:22.328938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:22.329205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.329205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:22.329929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.329929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:22.329929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:22.329929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:22.329929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:22.329929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:22.329929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.329929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:22.329929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:22.329929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.329929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:22.329929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:22.329929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.329929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:22.329929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:22.329929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:22.330249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:22.330790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:22.330790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:22.330790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:26.834176 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:26.836489 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:26.836489 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:26.836489 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:26.897336 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:26.897507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:26.899659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:26.899659 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:26.899659 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:26.899659 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:26.899659 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:26.899659 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:26.899659 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:26.899659 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:26.899659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.063 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:26.899659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:26.899659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:26.899659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:26.899659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:26.899659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:26.899659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:26.900489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:26.900489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:26.900489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:26.900489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:26.900489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:26.900489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:26.900489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:26.900489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:26.900489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:26.900489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:26.900489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:26.900489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:26.900489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:26.900489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:26.900886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:26.900886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:26.900886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:26.900886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:26.900886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:26.900886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:26.900886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:26.900886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:26.900886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:26.900886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:26.900886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:26.900886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:26.900886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:26.900886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:26.901334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:26.902340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:26.902340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:26.902340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:26.902340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:26.902340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:26.902340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:26.902340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:26.902340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:26.902340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:27.043378 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:27.043748 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:27.043894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:27.056652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 01 11:28:27.056948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Number of inconsistencies found at create/update: networks=17, security_groups=5, routers=11, router_ports=11 {{(pid=89427) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 01 11:28:27.057073 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=7 {{(pid=89427) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 01 11:28:27.057147 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 0885312d-0b5f-4eb8-b098-2739d84e7d46 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.107873 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-0885312d-0b5f-4eb8-b098-2739d84e7d46, resource={'id': '0885312d-0b5f-4eb8-b098-2739d84e7d46', 'name': 'tempest-AttachSCSIVolumeTestJSON-670951259-network', 'tenant_id': 'b0cb77273fab44d6849f7dd8241e2ca4', 'project_id': 'b0cb77273fab44d6849f7dd8241e2ca4', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['2255c7a1-3e6a-4462-bca5-1ef3783b79d4'], 'standard_attr_id': 164, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:26:16Z', 'updated_at': '2026-06-01T11:26:18Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 48319}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.108206 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-0885312d-0b5f-4eb8-b098-2739d84e7d46, col_values=(('external_ids', {'neutron:network_name': 'tempest-AttachSCSIVolumeTestJSON-670951259-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.108634 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-3e39979e-715a-44ea-aa61-a6fff3802d9f, external_ids_update=None, columns={'tag': 48319}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.108804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.110149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.110149 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.110149 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.110149 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.110149 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.110149 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.110149 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.110149 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.112646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 0885312d-0b5f-4eb8-b098-2739d84e7d46 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.112646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.112646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.112646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.112646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.112646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.112646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.112646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.112646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.112646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.112646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.113492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.113492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.113492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.113492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.113492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.113492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.113492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.113492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.113492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.113492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.113492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.113492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.113492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.113492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.114091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.114091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.114091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.114091 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 0d4134e5-3b3a-49d4-a466-c1b6f993b0ab (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.158933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-0d4134e5-3b3a-49d4-a466-c1b6f993b0ab, resource={'id': '0d4134e5-3b3a-49d4-a466-c1b6f993b0ab', 'name': 'private', 'tenant_id': '43b6be50ae544caeb6a73eced65fb6a7', 'project_id': '43b6be50ae544caeb6a73eced65fb6a7', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['16527f37-b4ae-44e5-9382-abb5e59d1cad', 'a33d59fc-0fd4-45c5-9fff-82704a1922a6'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:23:10Z', 'updated_at': '2026-06-01T11:23:13Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 48058}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.160573 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-0d4134e5-3b3a-49d4-a466-c1b6f993b0ab, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.160573 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-bedd3a6a-831e-4f6a-adf7-0842bd968bac, external_ids_update=None, columns={'tag': 48058}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.160573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.160777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.160777 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.160777 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.160777 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.160777 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.160777 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.160777 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.160777 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.161789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 0d4134e5-3b3a-49d4-a466-c1b6f993b0ab (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.161789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.161789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.161789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.161789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.161789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.161789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.161789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.161789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.161789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.161789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.162264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.162264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.162264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.162264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.162264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.162264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.162264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.162264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.162264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.162264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.162264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.162264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.162264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.162264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.162568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.162568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.162568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.162568 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 11a485ca-e8cf-4e4a-b33d-8efc6a0c7fd5 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.224749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-11a485ca-e8cf-4e4a-b33d-8efc6a0c7fd5, resource={'id': '11a485ca-e8cf-4e4a-b33d-8efc6a0c7fd5', 'name': 'tempest-device-tagging-net2-793095478', 'tenant_id': '0152ec13f36c471fb09635412aa139c5', 'project_id': '0152ec13f36c471fb09635412aa139c5', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['17fa54c9-2ffa-4180-950b-0b6643a12ea0'], 'standard_attr_id': 262, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:26:44Z', 'updated_at': '2026-06-01T11:26:50Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 58639}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.224749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-11a485ca-e8cf-4e4a-b33d-8efc6a0c7fd5, col_values=(('external_ids', {'neutron:network_name': 'tempest-device-tagging-net2-793095478', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.225124 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-d019f1d3-5e75-462a-8e48-29d6c19e011e, external_ids_update=None, columns={'tag': 58639}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.225124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.225124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.225124 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.225124 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.225124 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.225124 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.225124 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.225124 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.225124 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.233464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 11a485ca-e8cf-4e4a-b33d-8efc6a0c7fd5 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.233464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.233464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.233464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.233464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.233464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.233464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.233464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.233464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.233464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.233464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.233957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.233957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.233957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.233957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.233957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.233957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.233957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.233957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.233957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.233957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.233957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.233957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.233957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.233957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.234349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.234349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.234349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.234349 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 18af61c6-969d-4bbb-bc53-ce2313ba1de4 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.261641 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-18af61c6-969d-4bbb-bc53-ce2313ba1de4, resource={'id': '18af61c6-969d-4bbb-bc53-ce2313ba1de4', 'name': 'tempest-TaggedBootDevicesTest-2094608292-network', 'tenant_id': '0152ec13f36c471fb09635412aa139c5', 'project_id': '0152ec13f36c471fb09635412aa139c5', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['f53e824a-00f3-4021-b42f-1005bfd1ad23'], 'standard_attr_id': 48, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:25:51Z', 'updated_at': '2026-06-01T11:25:54Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 5583}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.261641 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-18af61c6-969d-4bbb-bc53-ce2313ba1de4, col_values=(('external_ids', {'neutron:network_name': 'tempest-TaggedBootDevicesTest-2094608292-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.262012 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-67a190ff-feb0-4d6f-ae40-5731299ab834, external_ids_update=None, columns={'tag': 5583}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.262012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.262726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.262726 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.262726 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.262726 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.262726 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.262726 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.262726 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.262726 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.263777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 18af61c6-969d-4bbb-bc53-ce2313ba1de4 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.263777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.263777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.263777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.263777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.263777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.263777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.263777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.263777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.263777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.263777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.264115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.264115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.264115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.264115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.264115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.264115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.264115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.264115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.264115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.264115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.264115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.264115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.264115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.264115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.264370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.264370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.264370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.264370 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 19da64d5-62e3-4b38-84f1-7d0971806ce9 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.279802 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:27.293517 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-19da64d5-62e3-4b38-84f1-7d0971806ce9, resource={'id': '19da64d5-62e3-4b38-84f1-7d0971806ce9', 'name': 'shared', 'tenant_id': '43b6be50ae544caeb6a73eced65fb6a7', 'project_id': '43b6be50ae544caeb6a73eced65fb6a7', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['c855657f-8ba2-422e-88fd-c5a71ec03ed7'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:24:23Z', 'updated_at': '2026-06-01T11:24:23Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 19987}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.293517 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-19da64d5-62e3-4b38-84f1-7d0971806ce9, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.293872 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-cb9dd134-19dd-491b-aafb-e4edb81fcf60, external_ids_update=None, columns={'tag': 19987}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.293872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.293872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.293872 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.293872 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.293872 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.293872 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.293872 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.293872 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.293872 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.294989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 19da64d5-62e3-4b38-84f1-7d0971806ce9 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.294989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.294989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.294989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.294989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.294989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.294989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.294989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.294989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.294989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.294989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.295389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.295389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.295389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.295389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.295389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.295389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.295389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.295389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.295389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.295389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.295389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.295389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.295389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.295389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.295813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.295813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.295813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.295813 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 21b086ac-a4f9-4f46-815b-9ed5309b18c5 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.305705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:27.317188 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:27.325824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:27.331096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-21b086ac-a4f9-4f46-815b-9ed5309b18c5, resource={'id': '21b086ac-a4f9-4f46-815b-9ed5309b18c5', 'name': 'tempest-ServersNegativeTestJSON-500937636-network', 'tenant_id': '4a198270d2d2463db945dbfabb3ef88c', 'project_id': '4a198270d2d2463db945dbfabb3ef88c', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['e2c59f0d-ad26-4e76-a668-f50b5f6ea3eb'], 'standard_attr_id': 303, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:27:05Z', 'updated_at': '2026-06-01T11:27:05Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 44593}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.331096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-21b086ac-a4f9-4f46-815b-9ed5309b18c5, col_values=(('external_ids', {'neutron:network_name': 'tempest-ServersNegativeTestJSON-500937636-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.331353 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-38b64acb-bbb1-418c-b12e-353e2c5bfeb2, external_ids_update=None, columns={'tag': 44593}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.331353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.331353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.331353 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.331353 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.331353 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.331353 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.331353 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.331353 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.331353 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.331999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 21b086ac-a4f9-4f46-815b-9ed5309b18c5 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.331999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.331999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.331999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.331999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.331999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.331999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.331999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.331999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.331999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.331999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.332302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.332302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.332302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.332302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.332302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.332302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.332302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.332302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.332302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.332302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.332302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.332302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.332302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.332302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.332649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.332649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.332649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.332649 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 4611f400-5a1a-421b-86fb-52f66b8385af (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.390490 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-4611f400-5a1a-421b-86fb-52f66b8385af, resource={'id': '4611f400-5a1a-421b-86fb-52f66b8385af', 'name': 'tempest-tagged-attachments-test-net-1912985626', 'tenant_id': '3a8f261bbd0d420fa617f2c66ec87a29', 'project_id': '3a8f261bbd0d420fa617f2c66ec87a29', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['9687ae17-ca01-43a5-961c-8375edebb248'], 'standard_attr_id': 209, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:26:33Z', 'updated_at': '2026-06-01T11:26:35Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 40524}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.390490 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-4611f400-5a1a-421b-86fb-52f66b8385af, col_values=(('external_ids', {'neutron:network_name': 'tempest-tagged-attachments-test-net-1912985626', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.390948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-82b8b4d9-63e8-4d4c-b387-6362319cf52c, external_ids_update=None, columns={'tag': 40524}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.390948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.390948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.390948 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.390948 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.390948 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.390948 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.390948 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.390948 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.390948 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.392552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 4611f400-5a1a-421b-86fb-52f66b8385af (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.392552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.392552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.392552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.392552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.392552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.392552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.392552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.392552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.392552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.392552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.393343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.393343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.393343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.393343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.393343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.393343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.393343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.393343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.393343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.393343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.393343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.393343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.393343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.393343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.394121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.394121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.394121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.394121 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 5ad39a4c-0b30-42c0-83b6-bd79dd4dc6c3 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.463434 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-5ad39a4c-0b30-42c0-83b6-bd79dd4dc6c3, resource={'id': '5ad39a4c-0b30-42c0-83b6-bd79dd4dc6c3', 'name': 'tempest-AttachSCSIVolumeTestJSON-1443990152-network', 'tenant_id': '3e878dd4b4fb45b8b8746447d6ffb33c', 'project_id': '3e878dd4b4fb45b8b8746447d6ffb33c', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['fa05b320-4bc5-469c-b021-5952f593ae59'], 'standard_attr_id': 88, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:25:53Z', 'updated_at': '2026-06-01T11:25:56Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 41297}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.464080 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-5ad39a4c-0b30-42c0-83b6-bd79dd4dc6c3, col_values=(('external_ids', {'neutron:network_name': 'tempest-AttachSCSIVolumeTestJSON-1443990152-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.464624 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-77e9aca9-2c2c-4edf-8b3e-c39fca534e0a, external_ids_update=None, columns={'tag': 41297}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.465334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.469848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.469848 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.469848 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.469848 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.469848 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.469848 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.469848 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.469848 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.473623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 5ad39a4c-0b30-42c0-83b6-bd79dd4dc6c3 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.473623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.473623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.473623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.473623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.473623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.473623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.473623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.473623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.473623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.473623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.473990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.473990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.473990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.473990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.473990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.473990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.473990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.473990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.473990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.473990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.473990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.473990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.473990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.473990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.474383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.474383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.474383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.474383 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource a52e4538-4a9a-4329-ab57-369c50e61479 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.516020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-a52e4538-4a9a-4329-ab57-369c50e61479, resource={'id': 'a52e4538-4a9a-4329-ab57-369c50e61479', 'name': 'tempest-VolumesAssistedSnapshotsTest-1295463772-network', 'tenant_id': '4209247bd8f2459c9590d2f6714847ae', 'project_id': '4209247bd8f2459c9590d2f6714847ae', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['4b1df990-91ed-4401-ba17-fe66342f32fc'], 'standard_attr_id': 173, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:26:17Z', 'updated_at': '2026-06-01T11:26:19Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 12332}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.516474 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-a52e4538-4a9a-4329-ab57-369c50e61479, col_values=(('external_ids', {'neutron:network_name': 'tempest-VolumesAssistedSnapshotsTest-1295463772-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.516894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-5f7c280f-609f-437d-8393-c2ad6a0fb0c1, external_ids_update=None, columns={'tag': 12332}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.517516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.520201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.520201 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.520201 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.520201 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.520201 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.520201 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.520201 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.520201 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.521467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource a52e4538-4a9a-4329-ab57-369c50e61479 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.521467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.521467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.521467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.521467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.521467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.521467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.521467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.521467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.521467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.521467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.521737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.521737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.521737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.521737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.521737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.521737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.521737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.521737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.521737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.521737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.521737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.521737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.521737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.521737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.522057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.522057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.522057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.522057 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource ad9585bf-9219-4132-a22f-2e6de82165b9 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.557646 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-ad9585bf-9219-4132-a22f-2e6de82165b9, resource={'id': 'ad9585bf-9219-4132-a22f-2e6de82165b9', 'name': 'tempest-ServerActionsTestOtherB-1791545800-network', 'tenant_id': '9539782a4a19497296b846de5abe6b5a', 'project_id': '9539782a4a19497296b846de5abe6b5a', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['79b43b49-7e07-4a20-9dee-f5f3cc72ad28'], 'standard_attr_id': 64, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:25:52Z', 'updated_at': '2026-06-01T11:25:55Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 53147}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.558234 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-ad9585bf-9219-4132-a22f-2e6de82165b9, col_values=(('external_ids', {'neutron:network_name': 'tempest-ServerActionsTestOtherB-1791545800-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.558828 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-cdad2287-3f2d-4ab6-a052-ea907c1fe25a, external_ids_update=None, columns={'tag': 53147}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.559617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.560801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.560801 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.560801 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.560801 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.560801 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.560801 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.560801 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.560801 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.562015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource ad9585bf-9219-4132-a22f-2e6de82165b9 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.562015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.562015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.562015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.562015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.562015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.562015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.562015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.562015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.562015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.562015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.562344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.562344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.562344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.562344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.562344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.562344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.562344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.562344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.562344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.562344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.562344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.562344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.562344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.562344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.562758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.562758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.562758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.562758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource bace1df3-e3ae-4b4e-9b7a-97fb45c30c50 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.600702 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-bace1df3-e3ae-4b4e-9b7a-97fb45c30c50, resource={'id': 'bace1df3-e3ae-4b4e-9b7a-97fb45c30c50', 'name': 'tempest-VolumesAssistedSnapshotsTest-1789562406-network', 'tenant_id': '87e5a32061134311afe784c451e9ed22', 'project_id': '87e5a32061134311afe784c451e9ed22', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['1f93e2d7-e7b9-4dee-9cf6-26e47ca1f633'], 'standard_attr_id': 80, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:25:53Z', 'updated_at': '2026-06-01T11:25:55Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 10583}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.603215 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-bace1df3-e3ae-4b4e-9b7a-97fb45c30c50, col_values=(('external_ids', {'neutron:network_name': 'tempest-VolumesAssistedSnapshotsTest-1789562406-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.603820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-8e9bc6db-71d2-4db1-aec4-600e4f99f525, external_ids_update=None, columns={'tag': 10583}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.604063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.605136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.605136 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.605136 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.605136 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.605136 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.605136 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.605136 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.605136 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.606423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource bace1df3-e3ae-4b4e-9b7a-97fb45c30c50 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.606423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.606423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.606423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.606423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.606423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.606423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.606423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.606423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.606423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.606423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.606656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.606656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.606656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.606656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.606656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.606656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.606656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.606656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.606656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.606656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.606656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.606656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.606656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.606656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.606886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.606886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.606886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.606886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource c6c73189-2fc5-412b-9ed5-038efd7341bc (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.650275 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, resource={'id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'name': 'public', 'tenant_id': '43b6be50ae544caeb6a73eced65fb6a7', 'project_id': '43b6be50ae544caeb6a73eced65fb6a7', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['21571d58-7621-44df-bdae-bad06600904d', '2bd944db-99d5-4573-8fc8-07e7942a1537'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-01T11:23:15Z', 'updated_at': '2026-06-01T11:23:22Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.655603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.655905 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-4ecb0f57-2a36-41bc-88d1-5cf4f555b097, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.656228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.658508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.658508 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.658508 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.658508 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.658508 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.658508 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.658508 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.658508 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.659755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource c6c73189-2fc5-412b-9ed5-038efd7341bc (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.659755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.659755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.659755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.659755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.659755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.659755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.659755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.659755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.659755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.659755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.660117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.660117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.660117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.660117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.660117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.660117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.660117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.660117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.660117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.660117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.660117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.660117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.660117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.660117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.660449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.660449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.660449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.661111 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource c909060a-4cf2-4566-b04c-eb085278d8c9 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.702495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-c909060a-4cf2-4566-b04c-eb085278d8c9, resource={'id': 'c909060a-4cf2-4566-b04c-eb085278d8c9', 'name': 'tempest-TaggedAttachmentsTest-525527502-network', 'tenant_id': '3a8f261bbd0d420fa617f2c66ec87a29', 'project_id': '3a8f261bbd0d420fa617f2c66ec87a29', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['df0f6014-228e-4710-8b74-b407539d5899'], 'standard_attr_id': 56, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:25:52Z', 'updated_at': '2026-06-01T11:25:54Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 45051}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.702944 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-c909060a-4cf2-4566-b04c-eb085278d8c9, col_values=(('external_ids', {'neutron:network_name': 'tempest-TaggedAttachmentsTest-525527502-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.703304 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-19bb0c6a-a8a3-4397-9cd7-e639dfdc76de, external_ids_update=None, columns={'tag': 45051}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.703462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.704334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.704334 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.704334 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.704334 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.704334 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.704334 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.704334 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.704334 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.705710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource c909060a-4cf2-4566-b04c-eb085278d8c9 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.705710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.705710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.705710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.705710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.705710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.705710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.705710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.705710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.705710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.705710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.706201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.706201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.706201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.706201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.706201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.706201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.706201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.706201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.706201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.706201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.706201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.706201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.706201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.706201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.706549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.706549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.706549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.706549 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource dc5bd8e3-b137-4987-a5da-24259a532ff8 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.755922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-dc5bd8e3-b137-4987-a5da-24259a532ff8, resource={'id': 'dc5bd8e3-b137-4987-a5da-24259a532ff8', 'name': 'tempest-device-tagging-net1-1643153829', 'tenant_id': '0152ec13f36c471fb09635412aa139c5', 'project_id': '0152ec13f36c471fb09635412aa139c5', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['c4fb4a75-60c2-453a-ab21-73aa93e61a39'], 'standard_attr_id': 255, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:26:42Z', 'updated_at': '2026-06-01T11:26:49Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 13108}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.756445 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-dc5bd8e3-b137-4987-a5da-24259a532ff8, col_values=(('external_ids', {'neutron:network_name': 'tempest-device-tagging-net1-1643153829', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.757220 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-db4c238e-0d98-4e6b-953b-c2f50dab0fbf, external_ids_update=None, columns={'tag': 13108}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.757343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.758025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.758025 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.758025 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.758025 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.758025 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.758025 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.758025 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.758025 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.759597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource dc5bd8e3-b137-4987-a5da-24259a532ff8 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.759597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.759597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.759597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.759597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.759597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.759597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.759597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.759597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.759597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.759597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.760084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.760084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.760084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.760084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.760084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.760084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.760084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.760084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.760084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.760084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.760084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.760084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.760084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.760084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.760466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.760466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.760466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.760466 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource e220aab3-f243-4f64-882f-9f41b4ad9568 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.802362 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-e220aab3-f243-4f64-882f-9f41b4ad9568, resource={'id': 'e220aab3-f243-4f64-882f-9f41b4ad9568', 'name': 'tempest-VolumesAssistedSnapshotsTest-409523542-network', 'tenant_id': '74094c7790964114a5f08237bedb33c1', 'project_id': '74094c7790964114a5f08237bedb33c1', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['1eee6263-f7b1-432f-ab8d-4c691f9f7889'], 'standard_attr_id': 229, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:26:37Z', 'updated_at': '2026-06-01T11:26:39Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 29037}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.803170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-e220aab3-f243-4f64-882f-9f41b4ad9568, col_values=(('external_ids', {'neutron:network_name': 'tempest-VolumesAssistedSnapshotsTest-409523542-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.803170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-a41b596e-3484-49b0-9b6c-e12300f36904, external_ids_update=None, columns={'tag': 29037}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.803170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.803683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.803683 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.803683 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.803683 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.803683 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.803683 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.803683 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.803683 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.804780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource e220aab3-f243-4f64-882f-9f41b4ad9568 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.804780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.804780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.804780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.804780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.804780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.804780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.804780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.804780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.804780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.804780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.805372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.805372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.805372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.805372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.805372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.805372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.805372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.805372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.805372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.805372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.805372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.805372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.805372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.805372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.805739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.805739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.805739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.805739 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource e269cae8-7e71-42fb-acbd-a209931890ef (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.843041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-e269cae8-7e71-42fb-acbd-a209931890ef, resource={'id': 'e269cae8-7e71-42fb-acbd-a209931890ef', 'name': 'tempest-TaggedBootDevicesTest-748350925-network', 'tenant_id': 'ad44a24245b843a99211f50d34044317', 'project_id': 'ad44a24245b843a99211f50d34044317', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['50f0d9f2-cd2f-4b02-97ba-c9f796015977'], 'standard_attr_id': 136, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:26:14Z', 'updated_at': '2026-06-01T11:26:16Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 19595}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.843727 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-e269cae8-7e71-42fb-acbd-a209931890ef, col_values=(('external_ids', {'neutron:network_name': 'tempest-TaggedBootDevicesTest-748350925-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.843727 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-a7222f92-e3bf-4fb2-8698-939d9312ea35, external_ids_update=None, columns={'tag': 19595}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.843727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.844627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.844627 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.844627 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.844627 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.844627 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.844627 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.844627 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.844627 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.845310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource e269cae8-7e71-42fb-acbd-a209931890ef (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.845310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.845310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.845310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.845310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.845310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.845310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.845310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.845310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.845310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.845310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.845766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.845766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.845766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.845766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.845766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.845766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.845766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.845766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.845766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.845766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.845766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.845766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.845766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.845766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.846270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.846270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.846270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.846270 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource eb0650d3-5cb3-4b6e-8a5e-7952155af16f (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.906320 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-eb0650d3-5cb3-4b6e-8a5e-7952155af16f, resource={'id': 'eb0650d3-5cb3-4b6e-8a5e-7952155af16f', 'name': 'tempest-TaggedAttachmentsTest-1766159657-network', 'tenant_id': '05f52527600e48139327948860d2883d', 'project_id': '05f52527600e48139327948860d2883d', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['cef0aaca-1713-45d4-8a81-54ba848d1385'], 'standard_attr_id': 144, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:26:15Z', 'updated_at': '2026-06-01T11:26:17Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 62140}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.906632 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-eb0650d3-5cb3-4b6e-8a5e-7952155af16f, col_values=(('external_ids', {'neutron:network_name': 'tempest-TaggedAttachmentsTest-1766159657-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.906996 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-ef99af03-9bd6-4ff8-b001-abe2f7c8c809, external_ids_update=None, columns={'tag': 62140}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:27.907108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.907866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:27.907866 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.907866 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:27.907866 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:27.907866 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.907866 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:27.907866 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.907866 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:27.909136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource eb0650d3-5cb3-4b6e-8a5e-7952155af16f (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.909136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:27.909136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:27.909136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:27.909136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:27.909136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:27.909136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:28:27.909136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:27.909136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.909136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.909136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:27.909457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:27.909457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:27.909457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:27.909457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:27.909457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:27.909457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:27.909457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:27.909457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:27.909457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:27.909457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:27.909457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:27.909457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:27.909457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:27.909457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:27.909811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:27.909811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:27.909811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:27.909811 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 1ff2853d-441f-40b9-ad16-a43509ef0133 (type: security_groups) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:27.972159 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Successfully bumped revision number for resource 1ff2853d-441f-40b9-ad16-a43509ef0133 (type: security_groups) to 3 Jun 01 11:28:27.972727 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 41e8f363-9205-411a-8576-53d6f0cbf966 (type: security_groups) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:28.005346 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Successfully bumped revision number for resource 41e8f363-9205-411a-8576-53d6f0cbf966 (type: security_groups) to 3 Jun 01 11:28:28.005703 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 44de6ab7-c7d5-49a2-b38a-58c144dbb9b7 (type: security_groups) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:28.032128 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Successfully bumped revision number for resource 44de6ab7-c7d5-49a2-b38a-58c144dbb9b7 (type: security_groups) to 3 Jun 01 11:28:28.032695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 5d87e29d-75fd-48c7-9fef-e627ab6d53cc (type: security_groups) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:28.072650 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Successfully bumped revision number for resource 5d87e29d-75fd-48c7-9fef-e627ab6d53cc (type: security_groups) to 3 Jun 01 11:28:28.072650 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 9dbbca8d-4c6a-4037-afe3-67fbfe9edafe (type: security_groups) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:28.108507 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Successfully bumped revision number for resource 9dbbca8d-4c6a-4037-afe3-67fbfe9edafe (type: security_groups) to 3 Jun 01 11:28:28.108869 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 1fe8ab3a-66b5-4d45-b369-53d0a650973f (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:28.328832 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, resource={'id': '1fe8ab3a-66b5-4d45-b369-53d0a650973f', 'name': 'tempest-ServersNegativeTestJSON-500937636-router', 'project_id': '4a198270d2d2463db945dbfabb3ef88c', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.160'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::364'}], 'enable_snat': True}, 'gw_port_id': '1a8c81ec-dcb6-4d94-b45d-24fe28f180bb', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.160', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::364', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:27:06Z', 'updated_at': '2026-06-01T11:28:27Z', 'revision_number': 4, 'tenant_id': '4a198270d2d2463db945dbfabb3ef88c'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.329004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-1fe8ab3a-66b5-4d45-b369-53d0a650973f, columns={'external_ids': {'neutron:router_name': 'tempest-ServersNegativeTestJSON-500937636-router', 'neutron:revision_number': '4', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.330141 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "1a8c81ec-dcb6-4d94-b45d-24fe28f180bb"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.330277 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "1a8c81ec-dcb6-4d94-b45d-24fe28f180bb"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.330391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.333261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:28.333261 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:28.333261 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:28.333261 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:28.333261 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:28.333261 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:28.333261 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.333261 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:28.333261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router 1fe8ab3a-66b5-4d45-b369-53d0a650973f. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.333261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 1fe8ab3a-66b5-4d45-b369-53d0a650973f (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.333261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:28.333261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:28.333715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:28.333715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:28.333715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:28.333715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:28:28.333715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:28:28.333715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:28:28.333715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:28:28.333715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:28:28.333715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:28:28.333715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:28:28.333715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:28.333715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:28.333715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:28.334079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:28.334079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:28.334079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:28.334079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:28.334079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:28.334079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:28.334079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:28.334079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:28.334079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:28.334079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:28.334079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:28.334079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:28.334079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:28.334079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:28.334369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:28.334369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:28.334369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.334369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:28.334369 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 21537149-b4d5-4fe6-9a59-2cd7a59048e4 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:28.440133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, resource={'id': '21537149-b4d5-4fe6-9a59-2cd7a59048e4', 'name': 'tempest-TaggedBootDevicesTest-748350925-router', 'project_id': 'ad44a24245b843a99211f50d34044317', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.69'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::215'}], 'enable_snat': True}, 'gw_port_id': '18d2afed-700f-49cf-8116-32f3cd754635', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.69', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::215', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:26:17Z', 'updated_at': '2026-06-01T11:26:24Z', 'revision_number': 3, 'tenant_id': 'ad44a24245b843a99211f50d34044317'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.441606 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, columns={'external_ids': {'neutron:router_name': 'tempest-TaggedBootDevicesTest-748350925-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.441606 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "18d2afed-700f-49cf-8116-32f3cd754635"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.441606 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "18d2afed-700f-49cf-8116-32f3cd754635"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.441606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.441606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:28.441606 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:28.441606 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:28.441606 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:28.441858 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:28.441858 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:28.441858 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.441858 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:28.441858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router 21537149-b4d5-4fe6-9a59-2cd7a59048e4. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.443112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 21537149-b4d5-4fe6-9a59-2cd7a59048e4 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.443112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:28.443112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:28.443112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:28.443112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:28.443112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:28.443112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:28:28.443112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:28:28.443112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:28:28.443112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:28:28.443112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:28:28.443939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:28:28.443939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:28:28.443939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:28.443939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:28.443939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:28.443939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:28.443939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:28.443939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:28.443939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:28.443939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:28.443939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:28.443939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:28.443939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:28.443939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:28.444305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:28.444305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:28.444305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:28.444305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:28.444305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:28.444305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:28.444305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:28.444305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.444305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:28.444305 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 31445a58-14ab-4255-83dd-8d63b7030d94 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:28.544593 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, resource={'id': '31445a58-14ab-4255-83dd-8d63b7030d94', 'name': 'tempest-ServerActionsTestOtherB-1791545800-router', 'project_id': '9539782a4a19497296b846de5abe6b5a', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.4'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::1f8'}], 'enable_snat': True}, 'gw_port_id': 'da3fef6e-ba6a-4091-9e14-a82952dd8f20', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.4', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::1f8', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:25:59Z', 'updated_at': '2026-06-01T11:26:10Z', 'revision_number': 3, 'tenant_id': '9539782a4a19497296b846de5abe6b5a'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.544947 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, columns={'external_ids': {'neutron:router_name': 'tempest-ServerActionsTestOtherB-1791545800-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.545115 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "da3fef6e-ba6a-4091-9e14-a82952dd8f20"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.545689 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "da3fef6e-ba6a-4091-9e14-a82952dd8f20"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.545689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.546143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:28.546143 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:28.546143 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:28.546143 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:28.546143 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:28.546143 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:28.546143 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.546143 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:28.546421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router 31445a58-14ab-4255-83dd-8d63b7030d94. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.549013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 31445a58-14ab-4255-83dd-8d63b7030d94 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.549013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:28.549013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:28.549013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:28.549013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:28.549013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:28.549013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:28:28.549013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:28:28.549013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:28:28.549013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:28:28.549013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:28:28.549682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:28:28.549682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:28:28.549682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:28.549682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:28.549682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:28.549682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:28.549682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:28.549682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:28.549682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:28.549682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:28.549682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:28.549682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:28.549682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:28.549682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:28.550234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:28.550234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:28.550234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:28.550234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:28.550234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:28.550234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:28.550234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:28.550234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.550234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:28.550234 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 3b91e059-e49f-4514-8d70-4177ecb4b4d8 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:28.676734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, resource={'id': '3b91e059-e49f-4514-8d70-4177ecb4b4d8', 'name': 'tempest-AttachSCSIVolumeTestJSON-670951259-router', 'project_id': 'b0cb77273fab44d6849f7dd8241e2ca4', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.113'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::194'}], 'enable_snat': True}, 'gw_port_id': 'f69c5732-2f48-41f8-b81a-9bb52dbdd456', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.113', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::194', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:26:22Z', 'updated_at': '2026-06-01T11:26:31Z', 'revision_number': 3, 'tenant_id': 'b0cb77273fab44d6849f7dd8241e2ca4'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.677041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, columns={'external_ids': {'neutron:router_name': 'tempest-AttachSCSIVolumeTestJSON-670951259-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.677255 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "f69c5732-2f48-41f8-b81a-9bb52dbdd456"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.677385 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "f69c5732-2f48-41f8-b81a-9bb52dbdd456"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.677488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.678227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:28.678227 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:28.678227 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:28.678227 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:28.678227 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:28.678227 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:28.678227 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.678227 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:28.678503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router 3b91e059-e49f-4514-8d70-4177ecb4b4d8. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.681908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 3b91e059-e49f-4514-8d70-4177ecb4b4d8 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.681908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:28.681908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:28.681908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:28.681908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:28.681908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:28.681908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:28:28.681908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:28:28.681908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:28:28.681908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:28:28.681908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:28:28.682380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:28:28.682380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:28:28.682380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:28.682380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:28.682380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:28.682380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:28.682380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:28.682380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:28.682380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:28.682380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:28.682380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:28.682380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:28.682380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:28.682380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:28.682696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:28.682696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:28.682696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:28.682696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:28.682696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:28.682696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:28.682696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:28.682696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.682696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:28.682696 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 6e25944c-6076-4d8d-bfcc-1f2feba96921 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:28.778428 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, resource={'id': '6e25944c-6076-4d8d-bfcc-1f2feba96921', 'name': 'tempest-VolumesAssistedSnapshotsTest-1295463772-router', 'project_id': '4209247bd8f2459c9590d2f6714847ae', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.36'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::47'}], 'enable_snat': True}, 'gw_port_id': 'e4f97c4f-c3c2-4128-b88a-c5e114338f18', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.36', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::47', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:26:24Z', 'updated_at': '2026-06-01T11:26:33Z', 'revision_number': 3, 'tenant_id': '4209247bd8f2459c9590d2f6714847ae'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.778643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, columns={'external_ids': {'neutron:router_name': 'tempest-VolumesAssistedSnapshotsTest-1295463772-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.778797 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "e4f97c4f-c3c2-4128-b88a-c5e114338f18"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.778930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "e4f97c4f-c3c2-4128-b88a-c5e114338f18"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.779056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.779523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:28.779523 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:28.779523 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:28.779523 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:28.779523 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:28.779523 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:28.779523 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.779523 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:28.779729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router 6e25944c-6076-4d8d-bfcc-1f2feba96921. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.781340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 6e25944c-6076-4d8d-bfcc-1f2feba96921 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.781340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:28.781340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:28.781340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:28.781340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:28.781340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:28.781340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:28:28.781340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:28:28.781340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:28:28.781340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:28:28.781340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:28:28.781680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:28:28.781680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:28:28.781680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:28.781680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:28.781680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:28.781680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:28.781680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:28.781680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:28.781680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:28.781680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:28.781680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:28.781680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:28.781680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:28.781680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:28.782097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:28.782097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:28.782097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:28.782097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:28.782097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:28.782097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:28.782097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:28.782097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.782097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:28.782097 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 7750344c-f24a-4234-96e5-d26a59b9fb13 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:28.940349 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, resource={'id': '7750344c-f24a-4234-96e5-d26a59b9fb13', 'name': 'tempest-AttachSCSIVolumeTestJSON-1443990152-router', 'project_id': '3e878dd4b4fb45b8b8746447d6ffb33c', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.17'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::173'}], 'enable_snat': True}, 'gw_port_id': '17a031b8-acae-4000-b4bd-e3c37217c0ad', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.17', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::173', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:26:02Z', 'updated_at': '2026-06-01T11:26:12Z', 'revision_number': 3, 'tenant_id': '3e878dd4b4fb45b8b8746447d6ffb33c'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.941104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, columns={'external_ids': {'neutron:router_name': 'tempest-AttachSCSIVolumeTestJSON-1443990152-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.941104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "17a031b8-acae-4000-b4bd-e3c37217c0ad"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.941104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "17a031b8-acae-4000-b4bd-e3c37217c0ad"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:28.942567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.942567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:28.942567 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:28.942567 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:28.942567 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:28.942567 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:28.942567 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:28.942567 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.942567 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:28.942567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router 7750344c-f24a-4234-96e5-d26a59b9fb13. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.946123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 7750344c-f24a-4234-96e5-d26a59b9fb13 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.946123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:28.946123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:28.946123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:28.946123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:28.946123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:28.946123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:28:28.946123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:28:28.946123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:28:28.946123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:28:28.946123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:28:28.946497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:28:28.946497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:28:28.946497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:28.946497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:28.946497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:28.946497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:28.946497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:28.946497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:28.946497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:28.946497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:28.946497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:28.946497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:28.946497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:28.946497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:28.946918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:28.946918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:28.946918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:28.946918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:28.946918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:28.946918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:28.946918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:28.946918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:28.946918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:28.946918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 80d43aaf-af36-4673-b1a6-ad3048a62b71 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:29.094857 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, resource={'id': '80d43aaf-af36-4673-b1a6-ad3048a62b71', 'name': 'tempest-TaggedAttachmentsTest-1766159657-router', 'project_id': '05f52527600e48139327948860d2883d', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.65'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::d6'}], 'enable_snat': True}, 'gw_port_id': 'd0ed35c8-82fb-4f4c-9991-d2e2c7d905cf', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.65', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::d6', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:26:19Z', 'updated_at': '2026-06-01T11:26:29Z', 'revision_number': 3, 'tenant_id': '05f52527600e48139327948860d2883d'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.095116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, columns={'external_ids': {'neutron:router_name': 'tempest-TaggedAttachmentsTest-1766159657-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.095386 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "d0ed35c8-82fb-4f4c-9991-d2e2c7d905cf"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.095453 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "d0ed35c8-82fb-4f4c-9991-d2e2c7d905cf"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.095569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.096245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:29.096245 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:29.096245 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:29.096245 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:29.096245 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:29.096245 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:29.096245 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.096245 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:29.099890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router 80d43aaf-af36-4673-b1a6-ad3048a62b71. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.099890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 80d43aaf-af36-4673-b1a6-ad3048a62b71 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.099890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:29.099890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:29.099890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:29.099890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:29.099890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:29.099890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:28:29.099890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:28:29.100445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:28:29.100445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:28:29.100445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:28:29.100445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:28:29.100445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:28:29.100445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:29.100445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:29.100445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:29.100445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:29.100445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:29.100445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:29.100445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:29.100445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:29.100445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:29.100903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:29.100903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:29.100903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:29.100903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:29.100903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:29.100903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:29.100903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:29.100903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:29.100903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:29.100903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:29.100903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.100903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:29.101256 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource cf57772b-a1f3-48fe-bdcf-a248e5e472b3 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:29.241951 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, resource={'id': 'cf57772b-a1f3-48fe-bdcf-a248e5e472b3', 'name': 'tempest-TaggedAttachmentsTest-525527502-router', 'project_id': '3a8f261bbd0d420fa617f2c66ec87a29', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.170'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::38c'}], 'enable_snat': True}, 'gw_port_id': '86423414-9b62-49f4-865b-039bd23e59a3', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.170', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::38c', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:25:58Z', 'updated_at': '2026-06-01T11:26:08Z', 'revision_number': 3, 'tenant_id': '3a8f261bbd0d420fa617f2c66ec87a29'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.243614 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, columns={'external_ids': {'neutron:router_name': 'tempest-TaggedAttachmentsTest-525527502-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.243614 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "86423414-9b62-49f4-865b-039bd23e59a3"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.243614 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "86423414-9b62-49f4-865b-039bd23e59a3"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.243614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.243614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:29.243614 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:29.243614 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:29.243614 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:29.244020 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:29.244020 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:29.244020 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.244020 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:29.244020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router cf57772b-a1f3-48fe-bdcf-a248e5e472b3. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.246261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource cf57772b-a1f3-48fe-bdcf-a248e5e472b3 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.246261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:29.246261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:29.246261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:29.246261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:29.246261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:29.246261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:28:29.246261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:28:29.246261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:28:29.246261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:28:29.246261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:28:29.247079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:28:29.247079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:28:29.247079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:29.247079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:29.247079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:29.247079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:29.247079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:29.247079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:29.247079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:29.247079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:29.247079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:29.247079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:29.247079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:29.247079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:29.247428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:29.247428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:29.247428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:29.247428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:29.247428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:29.247428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:29.247428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:29.247428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.247428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:29.247428 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource e9c21cb7-e55e-4621-97e3-882ef2c1c807 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:29.387922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, resource={'id': 'e9c21cb7-e55e-4621-97e3-882ef2c1c807', 'name': 'tempest-VolumesAssistedSnapshotsTest-409523542-router', 'project_id': '74094c7790964114a5f08237bedb33c1', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.179'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::3aa'}], 'enable_snat': True}, 'gw_port_id': '9f098f8e-6b6f-427c-b314-a5c2876ddba1', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.179', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::3aa', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:26:42Z', 'updated_at': '2026-06-01T11:26:47Z', 'revision_number': 3, 'tenant_id': '74094c7790964114a5f08237bedb33c1'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.388229 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, columns={'external_ids': {'neutron:router_name': 'tempest-VolumesAssistedSnapshotsTest-409523542-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.388407 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "9f098f8e-6b6f-427c-b314-a5c2876ddba1"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.388601 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "9f098f8e-6b6f-427c-b314-a5c2876ddba1"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.388747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.389365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:29.389365 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:29.389365 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:29.389365 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:29.389365 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:29.389365 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:29.389365 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.389365 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:29.389772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router e9c21cb7-e55e-4621-97e3-882ef2c1c807. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.391140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource e9c21cb7-e55e-4621-97e3-882ef2c1c807 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.391140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:29.391140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:29.391140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:29.391140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:29.391140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:29.391140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:28:29.391140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:28:29.391140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:28:29.391140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:28:29.391140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:28:29.391741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:28:29.391741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:28:29.391741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:29.391741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:29.391741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:29.391741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:29.391741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:29.391741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:29.391741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:29.391741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:29.391741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:29.391741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:29.391741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:29.391741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:29.392357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:29.392357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:29.392357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:29.392357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:29.392357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:29.392357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:29.392357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:29.392357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.392357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:29.392357 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource f50cd413-169b-4f93-a71c-90c58cef4cce (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:29.525994 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, resource={'id': 'f50cd413-169b-4f93-a71c-90c58cef4cce', 'name': 'tempest-VolumesAssistedSnapshotsTest-1789562406-router', 'project_id': '87e5a32061134311afe784c451e9ed22', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.163'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::167'}], 'enable_snat': True}, 'gw_port_id': '330b735f-303c-4832-8b3b-1d8485a3cbd7', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.163', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::167', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:26:01Z', 'updated_at': '2026-06-01T11:26:12Z', 'revision_number': 3, 'tenant_id': '87e5a32061134311afe784c451e9ed22'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.526356 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, columns={'external_ids': {'neutron:router_name': 'tempest-VolumesAssistedSnapshotsTest-1789562406-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.526567 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "330b735f-303c-4832-8b3b-1d8485a3cbd7"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.526567 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "330b735f-303c-4832-8b3b-1d8485a3cbd7"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.526740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.527188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:29.527188 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:29.527188 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:29.527188 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:29.527188 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:29.527188 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:29.527188 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.527188 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:29.527598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router f50cd413-169b-4f93-a71c-90c58cef4cce. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.529573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource f50cd413-169b-4f93-a71c-90c58cef4cce (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.529573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:29.529573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:29.529573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:29.529573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:29.529573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:29.529573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:28:29.529573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:28:29.529573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:28:29.529573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:28:29.529573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:28:29.530315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:28:29.530315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:28:29.530315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:29.530315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:29.530315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:29.530315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:29.530315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:29.530315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:29.530315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:29.530315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:29.530315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:29.530315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:29.530315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:29.530315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:29.531134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:29.531134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:29.531134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:29.531134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:29.531134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:29.531134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:29.531134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:29.531134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.531134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:29.531134 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource f7bee114-f44a-462c-b1b8-38a0e8f84bb1 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:29.644762 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, resource={'id': 'f7bee114-f44a-462c-b1b8-38a0e8f84bb1', 'name': 'tempest-TaggedBootDevicesTest-2094608292-router', 'project_id': '0152ec13f36c471fb09635412aa139c5', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.190'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::18'}], 'enable_snat': True}, 'gw_port_id': '2bd5f178-b9ed-4263-81a2-fe9c17971f46', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.190', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::18', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:25:56Z', 'updated_at': '2026-06-01T11:26:06Z', 'revision_number': 3, 'tenant_id': '0152ec13f36c471fb09635412aa139c5'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.645015 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, columns={'external_ids': {'neutron:router_name': 'tempest-TaggedBootDevicesTest-2094608292-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.645015 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "2bd5f178-b9ed-4263-81a2-fe9c17971f46"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.645015 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "2bd5f178-b9ed-4263-81a2-fe9c17971f46"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.645015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.645015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:29.645015 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:29.645015 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:29.645015 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:29.645345 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:29.645345 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:29.645345 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.645345 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:29.645345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router f7bee114-f44a-462c-b1b8-38a0e8f84bb1. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.645345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource f7bee114-f44a-462c-b1b8-38a0e8f84bb1 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.645345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:29.645345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:29.645345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:29.645345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:29.645345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:29.645770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:28:29.645770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:28:29.645770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:28:29.645770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:28:29.645770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:28:29.645770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:28:29.645770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:28:29.645770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:29.645770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:29.645770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:29.645770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:29.645770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:29.645770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:29.646169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:29.646169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:29.646169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:29.646169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:29.646169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:29.646169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:29.646169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:29.646169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:29.646169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:29.646169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:29.646169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:29.646169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:29.646169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:29.646575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.646575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:29.646575 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 17a031b8-acae-4000-b4bd-e3c37217c0ad (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:29.863255 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-17a031b8-acae-4000-b4bd-e3c37217c0ad, resource={'id': '17a031b8-acae-4000-b4bd-e3c37217c0ad', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:e7:c4:d5', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '7750344c-f24a-4234-96e5-d26a59b9fb13', 'device_owner': 'network:router_gateway', 'standard_attr_id': 120, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.17'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::173'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:26:02Z', 'updated_at': '2026-06-01T11:26:05Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.863495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-17a031b8-acae-4000-b4bd-e3c37217c0ad, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-7750344c-f24a-4234-96e5-d26a59b9fb13'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.17/24', '2001:db8::173/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.863818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=17a031b8-acae-4000-b4bd-e3c37217c0ad, lrouter_port=lrp-17a031b8-acae-4000-b4bd-e3c37217c0ad, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:29.864041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.866817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:29.866817 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:29.866817 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:29.866817 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:29.866817 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:29.866817 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:29.866817 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.866817 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:29.866817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 17a031b8-acae-4000-b4bd-e3c37217c0ad (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.866817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:29.866817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:29.866817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:29.866817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:29.866817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:29.867994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:28:29.867994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:29.867994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:29.867994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:29.867994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:29.867994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:29.867994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:29.867994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:29.867994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:29.867994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:29.867994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:29.867994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:29.867994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:29.868380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:29.868380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:29.868380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:29.868380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:29.868380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:29.868380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:29.868380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:29.868380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:29.868380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:29.868380 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 18d2afed-700f-49cf-8116-32f3cd754635 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:30.004775 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-18d2afed-700f-49cf-8116-32f3cd754635, resource={'id': '18d2afed-700f-49cf-8116-32f3cd754635', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:2d:b2:7d', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '21537149-b4d5-4fe6-9a59-2cd7a59048e4', 'device_owner': 'network:router_gateway', 'standard_attr_id': 179, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.69'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::215'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:26:18Z', 'updated_at': '2026-06-01T11:26:23Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:30.006744 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-18d2afed-700f-49cf-8116-32f3cd754635, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.69/24', '2001:db8::215/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:30.007181 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=18d2afed-700f-49cf-8116-32f3cd754635, lrouter_port=lrp-18d2afed-700f-49cf-8116-32f3cd754635, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:30.007406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:30.007931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:30.007931 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:30.007931 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:30.007931 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:30.007931 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:30.007931 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:30.007931 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:30.007931 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:30.009452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 18d2afed-700f-49cf-8116-32f3cd754635 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:30.009452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:30.009452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:30.009452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:30.009452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:30.009452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:30.009452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:28:30.009452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:30.009452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:30.009452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:30.009452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:30.010278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:30.010278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:30.010278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:30.010278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:30.010278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:30.010278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:30.010278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:30.010278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:30.010278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:30.010278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:30.010278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:30.010278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:30.010278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:30.010278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:30.010724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:30.010724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:30.010724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:30.010724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 1a8c81ec-dcb6-4d94-b45d-24fe28f180bb (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:30.030818 np0000001841 neutron-ovn-maintenance-worker[89427]: WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Skip fixing resource 1a8c81ec-dcb6-4d94-b45d-24fe28f180bb (type: router_ports). Resource does not exist in Neutron database anymore: neutron_lib.exceptions.PortNotFound: Port 1a8c81ec-dcb6-4d94-b45d-24fe28f180bb could not be found. Jun 01 11:28:30.031487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 2bd5f178-b9ed-4263-81a2-fe9c17971f46 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:30.192935 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-2bd5f178-b9ed-4263-81a2-fe9c17971f46, resource={'id': '2bd5f178-b9ed-4263-81a2-fe9c17971f46', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:8d:14:df', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'f7bee114-f44a-462c-b1b8-38a0e8f84bb1', 'device_owner': 'network:router_gateway', 'standard_attr_id': 110, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.190'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::18'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:25:57Z', 'updated_at': '2026-06-01T11:26:00Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:30.196289 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-2bd5f178-b9ed-4263-81a2-fe9c17971f46, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.190/24', '2001:db8::18/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:30.196289 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=2bd5f178-b9ed-4263-81a2-fe9c17971f46, lrouter_port=lrp-2bd5f178-b9ed-4263-81a2-fe9c17971f46, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:30.196289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:30.196289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:30.196289 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:30.196289 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:30.196289 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:30.196289 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:30.196289 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:30.196875 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:30.196875 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:30.196875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 2bd5f178-b9ed-4263-81a2-fe9c17971f46 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:30.196875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:30.196875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:30.196875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:30.196875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:30.196875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:30.196875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:28:30.196875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:30.196875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:30.196875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:30.197262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:30.197262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:30.197262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:30.197262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:30.197262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:30.197262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:30.197262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:30.197262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:30.197262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:30.197262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:30.197262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:30.197262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:30.197262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:30.197262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:30.197735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:30.197735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:30.197735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:30.197735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:30.197735 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 330b735f-303c-4832-8b3b-1d8485a3cbd7 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:30.379181 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-330b735f-303c-4832-8b3b-1d8485a3cbd7, resource={'id': '330b735f-303c-4832-8b3b-1d8485a3cbd7', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:b5:e8:ba', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'f50cd413-169b-4f93-a71c-90c58cef4cce', 'device_owner': 'network:router_gateway', 'standard_attr_id': 118, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.163'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::167'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:26:01Z', 'updated_at': '2026-06-01T11:26:08Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:30.379426 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-330b735f-303c-4832-8b3b-1d8485a3cbd7, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-f50cd413-169b-4f93-a71c-90c58cef4cce'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::167/64', '172.24.5.163/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:30.379682 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=330b735f-303c-4832-8b3b-1d8485a3cbd7, lrouter_port=lrp-330b735f-303c-4832-8b3b-1d8485a3cbd7, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:30.379865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:30.381714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:30.381714 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:30.381714 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:30.381714 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:30.381714 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:30.381714 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:30.381714 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:30.381714 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:30.381714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 330b735f-303c-4832-8b3b-1d8485a3cbd7 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:30.381714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:30.381714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:30.381714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:30.381714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:30.381714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:30.382087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:28:30.382087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:30.382087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:30.382087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:30.382087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:30.382087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:30.382087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:30.382087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:30.382087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:30.382087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:30.382087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:30.382087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:30.382087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:30.382291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:30.382291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:30.382291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:30.382291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:30.382291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:30.382291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:30.382291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:30.382291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:30.382291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:30.382291 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 86423414-9b62-49f4-865b-039bd23e59a3 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:30.884597 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-86423414-9b62-49f4-865b-039bd23e59a3, resource={'id': '86423414-9b62-49f4-865b-039bd23e59a3', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:53:13:0d', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'cf57772b-a1f3-48fe-bdcf-a248e5e472b3', 'device_owner': 'network:router_gateway', 'standard_attr_id': 112, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.170'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::38c'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:25:58Z', 'updated_at': '2026-06-01T11:26:02Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:30.885604 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-86423414-9b62-49f4-865b-039bd23e59a3, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::38c/64', '172.24.5.170/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:30.885604 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=86423414-9b62-49f4-865b-039bd23e59a3, lrouter_port=lrp-86423414-9b62-49f4-865b-039bd23e59a3, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:30.885604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:30.887530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:30.887530 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:30.887530 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:30.887530 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:30.887530 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:30.887530 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:30.887530 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:30.887530 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:30.892748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 86423414-9b62-49f4-865b-039bd23e59a3 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:30.892748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:30.892748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:30.892748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:30.892748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:30.892748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:30.892748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:28:30.892748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:30.892748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:30.892748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:30.892748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:30.893244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:30.893244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:30.893244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:30.893244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:30.893244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:30.893244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:30.893244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:30.893244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:30.893244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:30.893244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:30.893244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:30.893244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:30.893244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:30.893244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:30.893702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:30.893702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:30.893702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:30.893702 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 9f098f8e-6b6f-427c-b314-a5c2876ddba1 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:31.062440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-9f098f8e-6b6f-427c-b314-a5c2876ddba1, resource={'id': '9f098f8e-6b6f-427c-b314-a5c2876ddba1', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:57:5d:b5', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'e9c21cb7-e55e-4621-97e3-882ef2c1c807', 'device_owner': 'network:router_gateway', 'standard_attr_id': 254, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.179'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::3aa'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:26:42Z', 'updated_at': '2026-06-01T11:27:19Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.063178 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-9f098f8e-6b6f-427c-b314-a5c2876ddba1, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::3aa/64', '172.24.5.179/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.063178 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=9f098f8e-6b6f-427c-b314-a5c2876ddba1, lrouter_port=lrp-9f098f8e-6b6f-427c-b314-a5c2876ddba1, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.063178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.063662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:31.063662 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.063662 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:31.063662 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:31.063662 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.063662 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:31.063662 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.063662 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:31.064403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 9f098f8e-6b6f-427c-b314-a5c2876ddba1 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.064403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:31.064403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:31.064403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:31.064403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:31.064403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:31.064403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:28:31.064403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:31.064403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.064403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.064403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.064836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.064836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.064836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.064836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.064836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:31.064836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.064836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:31.064836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:31.064836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.064836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:31.064836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.064836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:31.064836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:31.064836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.065297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:31.065297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.065297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:31.065297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource d0ed35c8-82fb-4f4c-9991-d2e2c7d905cf (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:31.198446 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-d0ed35c8-82fb-4f4c-9991-d2e2c7d905cf, resource={'id': 'd0ed35c8-82fb-4f4c-9991-d2e2c7d905cf', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:46:6d:58', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '80d43aaf-af36-4673-b1a6-ad3048a62b71', 'device_owner': 'network:router_gateway', 'standard_attr_id': 191, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.65'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::d6'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:26:19Z', 'updated_at': '2026-06-01T11:26:24Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.198757 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-d0ed35c8-82fb-4f4c-9991-d2e2c7d905cf, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::d6/64', '172.24.5.65/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.199099 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=d0ed35c8-82fb-4f4c-9991-d2e2c7d905cf, lrouter_port=lrp-d0ed35c8-82fb-4f4c-9991-d2e2c7d905cf, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.199340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.199982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:31.199982 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.199982 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:31.199982 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:31.199982 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.199982 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:31.199982 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.199982 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:31.201617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource d0ed35c8-82fb-4f4c-9991-d2e2c7d905cf (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.201617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:31.201617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:31.201617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:31.201617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:31.201617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:31.201617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:28:31.201617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:31.201617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.201617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.201617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.202173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.202173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.202173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.202173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.202173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:31.202173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.202173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:31.202173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:31.202173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.202173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:31.202173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.202173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:31.202173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:31.202173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.202602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:31.202602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.202602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:31.202602 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource da3fef6e-ba6a-4091-9e14-a82952dd8f20 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:31.390474 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-da3fef6e-ba6a-4091-9e14-a82952dd8f20, resource={'id': 'da3fef6e-ba6a-4091-9e14-a82952dd8f20', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:08:b4:cd', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '31445a58-14ab-4255-83dd-8d63b7030d94', 'device_owner': 'network:router_gateway', 'standard_attr_id': 114, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.4'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::1f8'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:25:59Z', 'updated_at': '2026-06-01T11:26:05Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.390910 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-da3fef6e-ba6a-4091-9e14-a82952dd8f20, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-31445a58-14ab-4255-83dd-8d63b7030d94'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::1f8/64', '172.24.5.4/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.391115 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=da3fef6e-ba6a-4091-9e14-a82952dd8f20, lrouter_port=lrp-da3fef6e-ba6a-4091-9e14-a82952dd8f20, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.391341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.392404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:31.392404 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.392404 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:31.392404 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:31.392404 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.392404 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:31.392404 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.392404 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:31.393489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource da3fef6e-ba6a-4091-9e14-a82952dd8f20 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.393489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:31.393489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:31.393489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:31.393489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:31.393489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:31.393489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:28:31.393489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:31.393489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.393489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.393489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.393903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.393903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.393903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.393903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.393903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:31.393903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.393903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:31.393903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:31.393903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.393903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:31.393903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.393903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:31.393903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:31.393903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.394431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:31.394431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.394431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:31.394431 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource e4f97c4f-c3c2-4128-b88a-c5e114338f18 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:31.520803 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-e4f97c4f-c3c2-4128-b88a-c5e114338f18, resource={'id': 'e4f97c4f-c3c2-4128-b88a-c5e114338f18', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:8c:3a:8c', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '6e25944c-6076-4d8d-bfcc-1f2feba96921', 'device_owner': 'network:router_gateway', 'standard_attr_id': 200, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.36'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::47'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:26:25Z', 'updated_at': '2026-06-01T11:26:37Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.521098 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-e4f97c4f-c3c2-4128-b88a-c5e114338f18, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::47/64', '172.24.5.36/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.521401 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=e4f97c4f-c3c2-4128-b88a-c5e114338f18, lrouter_port=lrp-e4f97c4f-c3c2-4128-b88a-c5e114338f18, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.521804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.522083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:31.522083 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.522083 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:31.522083 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:31.522083 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.522083 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:31.522083 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.522083 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:31.522824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource e4f97c4f-c3c2-4128-b88a-c5e114338f18 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.522824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:31.522824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:31.522824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:31.522824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:31.522824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:31.522824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:28:31.522824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:31.522824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.522824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.522824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.523365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.523365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.523365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.523365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.523365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:31.523365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.523365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:31.523365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:31.523365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.523365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:31.523365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.523365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:31.523365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:31.523365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.523821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:31.523821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.523821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:31.523821 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource f69c5732-2f48-41f8-b81a-9bb52dbdd456 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:28:31.646064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-f69c5732-2f48-41f8-b81a-9bb52dbdd456, resource={'id': 'f69c5732-2f48-41f8-b81a-9bb52dbdd456', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:7b:94:20', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '3b91e059-e49f-4514-8d70-4177ecb4b4d8', 'device_owner': 'network:router_gateway', 'standard_attr_id': 196, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.113'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::194'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:26:22Z', 'updated_at': '2026-06-01T11:26:27Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.646296 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-f69c5732-2f48-41f8-b81a-9bb52dbdd456, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::194/64', '172.24.5.113/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.646571 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=f69c5732-2f48-41f8-b81a-9bb52dbdd456, lrouter_port=lrp-f69c5732-2f48-41f8-b81a-9bb52dbdd456, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.646789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.647348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:31.647348 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.647348 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:31.647348 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:31.647348 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.647348 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:31.647348 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.647348 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:31.648365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource f69c5732-2f48-41f8-b81a-9bb52dbdd456 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.648365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:31.648365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:28:31.648365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:28:31.648365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:28:31.648365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:28:31.648365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:28:31.648365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:31.648365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.648365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.648365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.648778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.648778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.648778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.648778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.648778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:31.648778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.648778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:31.648778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:31.648778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.648778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:31.648778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.648778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:31.648778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:31.648778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.649263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:31.649263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.649263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:31.649263 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 0a9b802d-fdcc-4593-b223-993d15fafeda (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:28:31.649263 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(0a9b802d-fdcc-4593-b223-993d15fafeda) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:28:31.657290 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 2e708d5d-5672-4c0f-8399-6534f801aab8 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:28:31.657644 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(2e708d5d-5672-4c0f-8399-6534f801aab8) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:28:31.663756 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 3453b93d-746a-487e-88fa-914fcc03c406 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:28:31.664119 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(3453b93d-746a-487e-88fa-914fcc03c406) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:28:31.669339 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 432e594a-0968-484a-af5c-e8b82a30552f (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:28:31.669638 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(432e594a-0968-484a-af5c-e8b82a30552f) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:28:31.675383 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 6a76eebd-4a78-47c1-920d-83a47f8d37ed (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:28:31.675751 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(6a76eebd-4a78-47c1-920d-83a47f8d37ed) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:28:31.682524 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 71d15d3f-ed97-402e-b0ba-3b414a869fa4 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:28:31.682925 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(71d15d3f-ed97-402e-b0ba-3b414a869fa4) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:28:31.690884 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource c73c0b83-c9d0-4cfc-9a0f-b53ff9b4b015 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:28:31.690884 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(c73c0b83-c9d0-4cfc-9a0f-b53ff9b4b015) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:28:31.697256 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_for_inconsistencies finished in 4.653 seconds Jun 01 11:28:31.697482 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:31.697593 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:31.698714 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.698864 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.699041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.699147 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.699249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.699345 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.699585 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.699722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.699858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.699997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.700129 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.700262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.700392 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.700523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.701084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:31.701084 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.701084 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:31.701084 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:31.701084 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.701084 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:31.701084 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.701084 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:31.702135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.702135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:31.702135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:31.702135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:31.702135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.702135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:31.702135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:31.702135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.702135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.702135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.702135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.702135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.702970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.702970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.702970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:31.702970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.702970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:31.702970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:31.702970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.702970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:31.702970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.702970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:31.702970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:31.702970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.702970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:31.703209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.703209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:31.703209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.703209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:31.703209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:31.703209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:31.703209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:31.703209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:31.703209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.703209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:31.703209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:31.703209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.703209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:31.703209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:31.703209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:31.703746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.704478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:31.704478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:31.704478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.704478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:31.704478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.704478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:31.704478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:31.704478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:31.704478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:28:31.704478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.704785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.704851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:31.704851 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.704851 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:31.704851 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:31.704851 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.704851 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:31.704851 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.704851 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:31.707451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.707451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:31.707451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:31.707451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:31.707451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.707451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:31.707451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.707451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.707451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.707451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:31.707451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.707451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:31.707451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.707451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:31.707451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:31.707941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.707941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:31.707941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.707941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:31.707941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:31.707941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.707941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:31.707941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.707941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:31.707941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.707941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:31.707941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:31.707941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:31.707941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.708219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:31.708219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:31.708219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:28:31.708219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:31.708219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:31.708219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.708219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.708219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.708219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.708219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.708219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.708219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.708219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:31.708450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.708450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:31.708450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:31.708450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.708450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:31.708450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.708450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:31.708450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:31.708450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.708450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:31.708450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.708450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:31.708748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.708748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:31.708748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:31.708748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:31.708748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:31.708748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:31.708748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.708748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:31.708748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:31.708748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.708748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:31.708748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:31.708748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.708748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:31.708748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:31.708748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.709083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:31.709374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.709374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:31.709374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:31.709374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.709374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:31.709374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.709374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:31.709374 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:31.709374 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:31.709374 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:31.709580 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.709580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.709580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:31.709580 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.709580 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:31.709580 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:31.709580 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.709580 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:31.709580 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.709580 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:31.710047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.710047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:31.710047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:31.710047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:31.710047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.710047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:31.710047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.710047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.710047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.710047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:31.710047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.710047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:31.710047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.710047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:31.710047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:31.710309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.710309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:31.710309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.710309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:31.710309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:31.710309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.710309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:31.710309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.710309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:31.710964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.710964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:31.710964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:31.710964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:31.710964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.710964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:31.710964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:31.710964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:31.710964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:31.710964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.710964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.711186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.711186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.711186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.711186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.711186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.711186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:31.711186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.711186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:31.711186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:31.711186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.711186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:31.711186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.711186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:31.711186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:31.711402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.711402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:31.711402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.711402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.712225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:31.712532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:31.712861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.712861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:31.712861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.712861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:31.712861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:31.712861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:31.712861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:31.713099 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.713345 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.713769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.714290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:31.714290 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.714290 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:31.714290 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:31.714290 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.714290 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:31.714290 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.714290 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:31.715270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.715270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:31.715270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:31.715270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:31.715270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.715270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:31.715270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:31.715270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.715270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.715270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.715270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.715270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.715663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.715663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.715663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:31.715663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.715663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:31.715663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:31.715663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.715663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:31.715663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.715663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:31.715663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:31.715663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.715663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:31.716062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.716062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:31.716416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.716416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:31.716416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:31.716416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:31.716416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:31.716416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:31.716416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.716416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:31.716416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:31.716416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.716416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:31.716416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:31.716416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.716416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:31.716416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:31.716416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:31.716738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.717061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:31.717061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.717061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:31.836256 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:31.836256 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:31.836256 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:31.836478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:31.899443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:31.899443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.899443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:31.899443 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.899443 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:31.899443 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:31.899443 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.899443 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:31.899443 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.899443 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:31.899443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.062 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.899443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:31.899443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:31.900508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:31.900508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.900508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:31.900508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:31.900508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.900508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.900508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.900508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.900508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.900508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:31.900508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.900508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:31.900508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.900508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:31.900855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:31.900855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.900855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:31.900855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.900855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:31.900855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:31.900855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.900855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:31.900855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.900855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:31.900855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.900855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:31.901191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:31.901643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:36.698402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:36.699007 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:36.699359 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:36.701149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.701521 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.701590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.701919 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.703136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.703136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.703136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.703136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.703136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.703334 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.703548 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7750344c-f24a-4234-96e5-d26a59b9fb13, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.704090 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-f50cd413-169b-4f93-a71c-90c58cef4cce, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.704090 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:36.704461 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=12): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.704461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.705225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:36.705225 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.705225 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:36.705225 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:36.705225 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.705225 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:36.705225 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.705225 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:36.706062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.706062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:36.706062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:36.706062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:36.706062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.706062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:36.706062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:36.706062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.706062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:36.706062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:36.706062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:36.706062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.706640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:36.706640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:36.706640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:36.706640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:36.706640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:36.706640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:36.706640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:36.706640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:36.706640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.706640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:36.706640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:36.706640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.706640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:36.707043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.707043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:36.707043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.707043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:36.707043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:36.707043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:36.707043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:36.707043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:36.707043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.707043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:36.707043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:36.707043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.707043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:36.707043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:36.707043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:36.707812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.708416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:36.708416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:36.708416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.708416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:36.708416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.708416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:36.708416 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:36.708416 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:36.708416 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:28:36.708416 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.708788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.708834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:36.708834 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.708834 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:36.708834 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:36.708834 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.708834 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:36.708834 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.708834 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:36.709533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.709533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:36.709533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:36.709533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:36.709533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.709533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:36.709533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:36.709533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:36.709533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.709533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:36.709533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:36.709533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:36.709533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:36.709533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:36.709533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:36.710089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:36.710089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:36.710089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.710089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:36.710089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:36.710089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.710089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:36.710089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.710089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:36.710089 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:36.710837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.710837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:36.710837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:36.710837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:36.710837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.710837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:36.710837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:36.710837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:28:36.710837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:36.710837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:36.710837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.710837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:36.711318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:36.711318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:36.711318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.711318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:36.711318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:36.711318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:36.711318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:36.711318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:36.711318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:36.711318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:36.711318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:36.711318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.711318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:36.711318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:36.711788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.711788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:36.711788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.711788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:36.711788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.711788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:36.711788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:36.711788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:36.711788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:36.711788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:36.711788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.711788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:36.711788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:36.711788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:36.712246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:36.712824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:36.712824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:36.712824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:36.712824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:36.712824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:36.712824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.712824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:36.712824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:36.712824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.712824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:36.712824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.712824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:36.712824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:36.712824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:36.713316 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:36.713316 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:36.713415 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.713662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.714188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:36.714188 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.714188 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:36.714188 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:36.714188 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.714188 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:36.714188 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.714188 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:36.714821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.714821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:36.714821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:36.714821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:36.714821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.714821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:36.714821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:36.714821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:36.714821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.714821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:36.714821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:36.714821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:36.714821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:36.714821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:36.714821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:36.715325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:36.715325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:36.715325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.715325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:36.715325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:36.715325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.715325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:36.715325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.715325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:36.715585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.715585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:36.715585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:36.715585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:36.715585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.715585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:36.715585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:36.715585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:36.715585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:36.715585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.715585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:36.715984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:36.715984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:36.715984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.715984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:36.715984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:36.715984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:36.715984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:36.715984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:36.715984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:36.715984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:36.715984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:36.715984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.715984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:36.715984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:36.716335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.716335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:36.716335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.716335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:36.716335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.716335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:36.716335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:36.716335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:36.716335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:36.716335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:36.716335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.716335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:36.716335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:36.716335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:36.716725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:36.717231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:36.717231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:36.717231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:36.717231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:36.717231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.717231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:36.717231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:36.717231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.717231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:36.717231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.717231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:36.717231 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:36.717231 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:36.717680 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:36.717680 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.718165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.718165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.718798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:36.718798 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.718798 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:36.718798 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:36.718798 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.718798 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:36.718798 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.718798 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:36.719453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.719453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:36.719453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:36.719453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:36.719453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.719453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:36.719453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:36.719453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.719453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:36.719453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:36.719453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:36.719453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.719792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:36.719792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:36.719792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:36.719792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:36.719792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:36.719792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:36.719792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:36.719792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:36.719792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.719792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:36.719792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:36.719792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.719792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:36.720175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.720175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:36.720175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.720175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:36.720175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:36.720175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:36.720175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:36.720175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:36.720175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.720175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:36.720175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:36.720175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.720175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:36.720175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:36.720175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:36.720725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.721334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:36.721334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:36.721334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.721334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:36.721334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.721334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:36.835805 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:36.836765 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:36.836765 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:36.838060 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:36.911572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:36.911572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.911832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:36.911832 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.911832 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:36.911832 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:36.911832 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.911832 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:36.911832 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.911832 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:36.912854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.075 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.912854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:36.912854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:36.912854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:36.912854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.912854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:36.912854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:36.912854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.912854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:36.912854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:36.912854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:36.912854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.913502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:36.913502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:36.913502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:36.913502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:36.913502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:36.913502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:36.913502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:36.913502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:36.913502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.913502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:36.913502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:36.913502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.913502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:36.913831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.913831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:36.913951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.913951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:36.913951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:36.913951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:36.913951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:36.913951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:36.913951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.913951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:36.913951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:36.913951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.913951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:36.913951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:36.913951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.913951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:36.913951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:36.913951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:36.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:36.915419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:36.915419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:36.915419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:41.698874 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:41.699898 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:41.699999 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:41.702615 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:41.702615 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:41.702615 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:41.702615 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3b91e059-e49f-4514-8d70-4177ecb4b4d8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:41.702615 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-6e25944c-6076-4d8d-bfcc-1f2feba96921, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:41.702987 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:41.702987 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:41.702987 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:41.702987 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:41.702987 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:41.703123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-cf57772b-a1f3-48fe-bdcf-a248e5e472b3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:41.703123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.704404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:41.704404 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.704404 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:41.704404 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:41.704404 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.704404 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:41.704404 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.704404 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:41.706895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.706895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:41.706895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:41.706895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:41.706895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.706895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:41.706895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:41.706895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.706895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:41.706895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:41.706895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:41.706895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.707289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:41.707289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:41.707289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:41.707289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:41.707289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:41.707289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:41.707289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:41.707289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:41.707289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.707289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:41.707289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:41.707289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.707289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:41.707607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.707607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:41.708114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.708114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:41.708114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:41.708114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:41.708114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:41.708114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:41.708114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.708114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:41.708114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:41.708114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.708114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:41.708114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:41.708114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.708114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:41.708114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:41.708114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:41.708512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.709034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:41.709034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.709034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:41.709034 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:41.709529 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:41.709529 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:41.709529 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:28:41.710424 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:41.710424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.710508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:41.710508 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.710508 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:41.710508 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:41.710508 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.710508 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:41.710508 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.710508 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:41.711297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.711297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:41.711297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:41.711297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:41.711297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.711297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:41.711297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:41.711297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:41.711297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.711297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:41.711297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:41.711297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:41.711297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:41.711297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:41.711297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:41.711648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:41.711648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:41.711648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.711648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:41.711648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:41.711648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.711648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:41.711648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.711648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:41.712013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.712013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:41.712013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:41.712013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:41.712013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.712013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:41.712013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:41.712013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:28:41.712013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:41.712013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:41.712013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.712013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:41.712397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:41.712397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:41.712397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.712397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:41.712397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:41.712397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:41.712397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:41.712397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:41.712397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:41.712397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:41.712397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:41.712397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.712397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:41.712397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:41.712815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.712815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:41.712815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.712815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:41.715547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.715547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:41.715547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:41.715547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:41.715547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:41.715547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:41.715547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.715547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:41.715547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:41.715547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.715547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:41.715547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:41.715547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.715547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:41.715547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:41.715547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:41.716180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:41.716509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.716509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:41.716509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:41.716509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.716509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:41.716509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.716509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:41.716509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:41.716509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:41.716509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:41.716734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:41.717518 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:41.717518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.717777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:41.718045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:41.718045 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.718045 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:41.718045 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:41.718045 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.718045 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:41.718045 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.718045 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:41.719588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.719588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:41.719588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:41.719588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:41.719588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.719588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:41.719588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:41.719588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:41.719588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.719588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:41.719588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:41.719588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:41.719588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:41.719588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:41.719588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:41.720021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:41.720021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:41.720021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.720021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:41.720021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:41.720021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.720021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:41.720021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.720021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:41.722635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.722635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:41.722635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:41.722635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:41.722635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.722635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:41.722635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:41.722635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:41.722635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:41.722635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.722635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:41.722948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:41.722948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:41.722948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.722948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:41.722948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:41.722948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:41.722948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:41.722948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:41.722948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:41.722948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:41.722948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:41.722948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.722948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:41.722948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:41.723196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.723196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:41.723196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.723196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.724180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:41.724659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:41.725230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.725230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:41.725230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.725230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:41.726672 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:41.726872 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:41.727161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:41.728116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:41.729937 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:41.729937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.729937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:41.729937 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.729937 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:41.729937 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:41.729937 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.729937 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:41.729937 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.729937 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:41.731100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.731100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:41.731100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:41.731100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:41.731100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.731100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:41.731100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:41.731100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.731100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:41.731100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:41.731100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:41.731100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.731371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:41.731371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:41.731371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:41.731371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:41.731371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:41.731371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:41.731371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:41.731371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:41.731371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.731371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:41.731371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:41.731371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.731371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:41.731626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.731626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:41.734745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.734745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:41.734745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:41.734745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:41.734745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:41.734745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:41.734745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.734745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:41.734745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:41.734745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.734745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:41.734745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:41.734745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.734745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:41.734745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:41.734745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:41.735232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.735946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:41.735946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.735946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:41.836881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:41.836881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:41.837112 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:41.837591 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:41.899555 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:41.899555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.899888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:41.899888 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.899888 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:41.899888 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:41.899888 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.899888 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:41.899888 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.899888 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:41.901133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.063 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.901133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:41.901133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:41.901133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:41.901133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.901133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:41.901133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:41.901133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.901133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:41.901133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:41.901133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:41.901133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.901666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:41.901666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:41.901666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:41.901666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:41.901666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:41.901666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:41.901666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:41.901666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:41.901666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.901666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:41.901666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:41.901666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.901666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:41.902389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.902389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:41.902782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.902782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:41.902782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:41.902782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:41.902782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:41.902782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:41.902782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.902782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:41.902782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:41.902782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.902782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:41.902782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:41.902782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.902782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:41.902782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:41.902782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:41.903225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:41.903615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:41.903615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:41.903615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:46.702093 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:46.702682 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:46.702682 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:46.705036 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:46.705254 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:46.705437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-80d43aaf-af36-4673-b1a6-ad3048a62b71, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:46.705612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e9c21cb7-e55e-4621-97e3-882ef2c1c807, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:46.705757 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:46.705929 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:46.706072 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:46.706180 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:46.706314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.707989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:46.707989 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.707989 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:46.707989 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:46.707989 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.707989 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:46.707989 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.707989 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:46.711021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.711021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:46.711021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:46.711021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:46.711021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.711021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:46.711021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:46.711021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.711021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:46.711021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:46.711021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:46.711021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.711370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:46.711370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:46.711370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:46.711370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:46.711370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:46.711370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:46.711370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:46.711370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:46.711370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.711370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:46.711370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:46.711370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.711370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:46.711646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.711646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:46.713098 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:46.714461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.714461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:46.714461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:46.714461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:46.714461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:46.714461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:46.714461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.714461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:46.714461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:46.714461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.714461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:46.714461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:46.714461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.714461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:46.714461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:46.714461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:46.715182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.715749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:46.715749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.715749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:46.715749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:46.715749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:46.715897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:28:46.716667 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:46.717793 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:46.718118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.719907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:46.719907 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.719907 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:46.719907 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:46.719907 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.719907 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:46.719907 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.719907 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:46.720591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.720591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:46.720591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:46.720591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:46.720591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.720591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:46.720591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:46.720591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:46.720591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.720591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:46.720591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:46.720591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:46.720591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:46.720591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:46.720591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:46.721234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:46.721234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:46.721234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.721234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:46.721234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:46.721234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.721234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:46.721234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.721234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:46.721943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.721943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:46.721943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:46.721943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:46.721943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.721943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:46.721943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:46.721943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:28:46.721943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:46.721943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:46.721943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.721943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:46.722718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:46.722718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:46.722718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.722718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:46.722718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:46.722718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:46.722718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:46.722718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:46.722718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:46.722718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:46.722718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:46.722718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.722718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:46.722718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:46.723993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.723993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:46.723993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.723993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:46.723993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.723993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:46.723993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:46.723993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:46.723993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:46.723993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:46.723993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.723993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:46.723993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:46.723993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:46.724449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:46.725068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:46.725068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:46.725068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:46.725068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:46.725068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:46.725068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.725068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:46.725068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:46.725068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.725068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:46.725068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.725068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:46.725068 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:46.725068 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:46.725507 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:46.725507 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:46.725507 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:46.725507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.727634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:46.727634 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.727634 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:46.727634 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:46.727634 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.727634 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:46.727634 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.727634 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:46.731745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.731745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:46.731745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:46.731745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:46.731745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.731745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:46.731745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:46.731745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:46.731745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.731745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:46.731745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:46.731745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:46.731745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:46.731745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:46.731745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:46.733205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:46.733205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:46.733205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.733205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:46.733205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:46.733205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.733205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:46.733205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.733205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:46.733205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.733205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:46.733205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:46.733205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:46.733205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.734222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:46.734222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:46.734222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:46.734222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:46.734222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.734222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:46.734222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:46.734222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:46.734222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.734222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:46.734222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:46.734222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:46.734222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:46.736123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:46.736123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:46.736123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:46.736123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:46.736123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.736123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:46.736123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:46.736123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.736123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:46.736123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.736123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:46.736123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:46.737620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.739208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:46.740123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:46.740123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:46.740123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:46.740123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:46.740123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:46.740123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.741894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:46.741894 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.741894 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:46.741894 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:46.741894 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.741894 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:46.741894 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.741894 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:46.742919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.742919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:46.742919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:46.742919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:46.742919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.742919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:46.742919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:46.742919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.742919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:46.742919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:46.742919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:46.742919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.743368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:46.743368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:46.743368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:46.743368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:46.743368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:46.743368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:46.743368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:46.743368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:46.743368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.743368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:46.743368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:46.743368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.743368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:46.744137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.744137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:46.744137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.744137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:46.744137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:46.744137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:46.744137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:46.744137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:46.744137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.744137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:46.744137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:46.744137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.744137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:46.744137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:46.744137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:46.744661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.745180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:46.745180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:46.745180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.745180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:46.745180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.745180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:46.839899 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:46.839899 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:46.839899 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:46.839899 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:46.923625 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:46.924371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.924941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:46.924941 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.924941 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:46.924941 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:46.924941 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.924941 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:46.924941 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.924941 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:46.927261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.087 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.927261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:46.927261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:46.927261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:46.927261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.927261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:46.927261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:46.927261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.927261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:46.927261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:46.927261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:46.927261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.928031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:46.928031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:46.928031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:46.928031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:46.928031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:46.928031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:46.928031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:46.928031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:46.928031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.928031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:46.928031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:46.928031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.928031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:46.928413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.928413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:46.928413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.928413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:46.928413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:46.928413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:46.928413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:46.928413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:46.928413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.928413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:46.928413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:46.928413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.928413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:46.928413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:46.928413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:46.928893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:46.929515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:46.929515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:46.929515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:46.929515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:46.929515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:46.929515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:51.702701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:51.704732 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:51.704732 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:51.706161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:51.706161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:51.707284 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:51.707284 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:51.707495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:51.707749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:51.708532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.712390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:51.712390 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.712390 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:51.712390 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:51.712390 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.712390 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:51.712390 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.712390 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:51.712390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.712390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:51.712390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:51.712390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:51.712390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.712390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:51.712390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:51.713043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.713043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:51.713043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:51.713043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:51.713043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.713043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:51.713043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:51.713043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:51.713043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:51.713043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:51.713043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:51.713043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:51.713043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:51.713043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.713500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:51.713500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:51.713500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.713500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:51.713500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.713500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:51.713500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.713500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:51.713500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:51.713500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:51.713500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:51.713500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:51.713500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.713500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:51.713984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:51.714635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:51.714635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:51.714635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:51.714635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.714635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:51.714635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:51.714635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.714635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:51.714635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.714635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:51.715625 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:51.716181 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:51.716861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:51.717102 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:28:51.717712 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:51.718221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.722427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:51.723211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:51.723211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:51.723211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:51.723211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:51.723211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:51.723211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:51.723211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:51.723211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.723211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:51.723211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:51.723211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.723211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:51.723211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.723211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:51.723211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.723713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:51.723713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:51.723713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:51.723713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.723713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:51.723713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:51.723713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:28:51.723713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:51.723713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:51.723713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.723713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:51.723713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:51.723713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:51.724196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.724196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:51.724196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:51.724196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:51.724196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:51.724196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:51.724196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:51.724196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:51.724196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:51.724196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.724196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:51.724196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:51.724196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.724196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:51.724681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.724681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:51.724681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.724681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:51.724681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:51.724681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:51.724681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:51.724681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:51.724681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.724681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:51.724681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:51.724681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.724681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:51.724681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:51.724681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:51.725197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:51.726217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:51.726217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:51.726217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.726217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:51.726217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:51.726217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.726217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:51.726217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.726217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:51.726217 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:51.728431 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:51.729823 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:51.730110 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:51.732234 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:51.734105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.735428 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:51.736056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:51.736056 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.736056 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:51.736056 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:51.736056 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.736056 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:51.736056 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.736056 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:51.738378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.738378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:51.738378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:51.738378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:51.738378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.738378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:51.738378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:51.738378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:51.738378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.738378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:51.738378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:51.738378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:51.738378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:51.738378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:51.738378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:51.738906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:51.738906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:51.738906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.738906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:51.738906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:51.738906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.738906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:51.738906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.738906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:51.739355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.739355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:51.739355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:51.739355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:51.739355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.739355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:51.739355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:51.739355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:51.739355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:51.739355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.739355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:51.739810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:51.739810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:51.739810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.739810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:51.739810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:51.739810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:51.739810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:51.739810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:51.739810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:51.739810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:51.739810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:51.739810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.739810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:51.739810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:51.740315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.740315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:51.740315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.740315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.743004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:51.743613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:51.744133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.744133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:51.744133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.744133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:51.744133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:51.745101 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:51.745325 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:51.746058 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:51.747061 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:51.749415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.752899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:51.752899 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.752899 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:51.752899 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:51.752899 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.752899 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:51.752899 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.752899 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:51.753741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.753741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:51.753741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:51.753741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:51.753741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.753741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:51.753741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:51.753741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.753741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:51.753741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:51.753741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:51.753741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.754052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:51.754052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:51.754052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:51.754052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:51.754052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:51.754052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:51.754052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:51.754052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:51.754052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.754052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:51.754052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:51.754052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.754052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:51.754377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.754377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:51.759288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.759288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:51.759288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:51.759288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:51.759288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:51.759288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:51.759288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.759288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:51.759288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:51.759288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.759288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:51.759288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:51.759288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.759288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:51.759288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:51.759288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:51.760661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.761329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:51.761329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.761329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:51.837926 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:51.838352 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:51.838469 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:51.839403 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:51.911357 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:51.911643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.912407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:51.912407 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.912407 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:51.912407 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:51.912407 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.912407 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:51.912407 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.912407 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:51.913743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.074 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.913743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:51.913743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:51.913743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:51.913743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.913743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:51.913743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:51.913743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.913743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:51.913743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:51.913743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:51.913743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.914014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:51.914014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:51.914014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:51.914014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:51.914014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:51.914014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:51.914014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:51.914014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:51.914014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.914014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:51.914014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:51.914014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.914014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:51.914244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.914244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:51.915169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.915169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:51.915169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:51.915169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:51.915169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:51.915169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:51.915169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.915169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:51.915169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:51.915169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.915169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:51.915169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:51.915169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.915169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:51.915169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:51.915169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:51.915493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:51.915822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:51.915822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:51.915822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:56.704647 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:56.706254 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:56.706254 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:56.707143 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:56.707563 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:56.707907 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:56.708262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:56.708692 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:56.709244 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:56.709693 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:56.710070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.710630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:56.710630 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.710630 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:56.710630 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:56.710630 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.710630 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:56.710630 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.710630 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:56.712745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.712745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:56.712745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:56.712745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:56.712745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.712745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:56.712745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:56.712745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.712745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:56.712745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:56.712745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:56.712745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.713088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:56.713088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:56.713088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:56.713088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:56.713088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:56.713088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:56.713088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:56.713088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:56.713088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.713088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:56.713088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:56.713088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.713088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:56.713346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.713346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:56.713346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.713346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:56.713346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:56.713346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:56.713346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:56.713346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:56.713346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.713346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:56.713346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:56.713346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.713346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:56.713346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:56.713346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:56.713673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.714034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:56.714034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:56.714034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.714034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:56.714034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.714034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:56.716176 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:56.716588 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:56.716848 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:56.717236 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:28:56.717551 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:56.718185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.718411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:56.718411 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.718411 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:56.718411 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:56.718411 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.718411 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:56.718411 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.718411 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:56.719484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.719484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:56.719484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:56.719484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:56.719484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.719484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:56.719484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:56.719484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:56.719484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.719484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:56.719484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:56.719484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:56.719484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:56.719484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:56.719484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:56.719867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:56.719867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:56.719867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.719867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:56.719867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:56.719867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.719867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:56.719867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.719867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:56.721062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.721062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:56.721062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:56.721062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:56.721062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.721062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:56.721062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:56.721062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:28:56.721062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:56.721062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:56.721062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.721062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:56.721870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:56.721870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:56.721870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.721870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:56.721870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:56.721870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:56.721870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:56.721870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:56.721870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:56.721870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:56.721870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:56.721870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.721870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:56.721870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:56.722131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.722131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:56.722131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.722131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:56.723203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.723203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:56.723203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:56.723203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:56.723203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:56.723203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:56.723203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.723203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:56.723203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:56.723203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.723203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:56.723203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:56.723203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.723203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:56.723203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:56.723203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:56.723967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:56.724453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.724453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:56.724453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:56.724453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.724453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:56.724453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.724453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:56.728506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:56.729173 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:56.729173 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:56.729173 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:56.729629 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:56.730263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.730512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:56.730512 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.730512 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:56.730512 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:56.730512 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.730512 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:56.730512 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.730512 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:56.731379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.731379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:28:56.731379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:56.731379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:56.731379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.731379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:56.731379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:56.731379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:28:56.731379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.731379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:28:56.731379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:56.731379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:28:56.731379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:56.731379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:28:56.731379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:28:56.732792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:56.732792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:28:56.732792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.732792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:28:56.732792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:28:56.732792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.732792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:28:56.732792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.732792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:28:56.732792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.732792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:56.732792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:56.732792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:56.732792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.733087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:56.733087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:56.733087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:56.733087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:56.733087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.733087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:56.733087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:56.733087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:56.733087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.733087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:56.733087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:56.733087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:56.733087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:56.733310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:56.733310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:56.733310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:56.733310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:56.733310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.733310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:56.733310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:56.733310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.733310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:56.733310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.733310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.734468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:56.734977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:56.735886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.735886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:56.735886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.735886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:56.743412 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:56.744123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:56.744123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:56.744123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:56.746723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:56.746723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:56.746723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.748036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:56.748036 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.748036 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:56.748036 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:56.748036 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.748036 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:56.748036 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.748036 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:56.749146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.749146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:56.749146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:56.749146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:56.749146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.749146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:56.749146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:56.749146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.749146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:56.749146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:56.749146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:56.749146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.749481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:56.749481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:56.749481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:56.749481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:56.749481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:56.749481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:56.749481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:56.749481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:56.749481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.749481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:56.749481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:56.749481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.749481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:56.749734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.749734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:56.750266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.750266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:56.750266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:56.750266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:56.750266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:56.750266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:56.750266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.750266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:56.750266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:56.750266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.750266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:56.750266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:56.750266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.750266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:56.750266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:56.750266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:56.750607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.750989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:56.750989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.750989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:28:56.838455 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:28:56.838913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:28:56.839086 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:28:56.840096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:28:56.908937 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:28:56.910325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.910606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:28:56.910606 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.910606 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:28:56.910606 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:28:56.910606 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.910606 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:28:56.910606 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.910606 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:28:56.911723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.072 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.911723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:28:56.911723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:56.911723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:28:56.911723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.911723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:56.911723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:56.911723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.911723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:56.911723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:56.911723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:28:56.911723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.912179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:28:56.912179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:56.912179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:28:56.912179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:56.912179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:28:56.912179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:28:56.912179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:56.912179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:28:56.912179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.912179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:28:56.912179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:28:56.912179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.912179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:28:56.912769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.912769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:28:56.912867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.912867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:28:56.912867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:28:56.912867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:28:56.912867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:28:56.912867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:28:56.912867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.912867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:28:56.912867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:28:56.912867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.912867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:28:56.912867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:28:56.912867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.912867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:28:56.912867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:28:56.912867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:28:56.913317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:28:56.913900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:28:56.913900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:28:56.913900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:01.705585 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:01.708770 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:01.708770 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:01.713777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:01.713904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:01.714112 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:01.714289 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:01.714449 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:01.714642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:01.714810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:01.714954 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:01.715104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.716500 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:01.717621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:01.717621 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.717621 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:01.717621 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:01.717621 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.717621 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:01.717621 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.717621 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:01.720857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.012 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.720857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:01.720857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:01.720857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:01.720857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.720857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:01.720857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:01.720857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.720857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:01.720857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:01.720857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:01.720857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.721440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:01.721440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:01.721440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:01.721440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:01.721440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:01.721440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:01.721440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:01.721440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:01.721440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.721440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:01.721440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:01.721440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.721440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:01.721904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.721904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:01.722287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.722287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:01.722287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:01.722287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:01.722287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:01.722287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:01.722287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.722287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:01.722287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:01.722287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.722287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:01.722287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:01.722287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.722287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:01.722287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:01.722287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:01.722926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.723493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:01.723493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.723493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:01.723666 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:01.723765 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:01.723886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:29:01.727304 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:01.727521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.728669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:01.728669 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.728669 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:01.728669 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:01.728669 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.728669 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:01.728669 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.728669 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:01.730417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.730417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:01.730417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:01.730417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:01.730417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.730417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:01.730417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:01.730417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:01.730417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.730417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:01.730417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:01.730417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:01.730417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:01.730417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:01.730417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:01.731105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:01.731105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:01.731105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.731105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:01.731105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:01.731105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.731105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:01.731105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.731105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:01.731366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.731366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:01.731366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:01.731366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:01.731366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.731366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:01.731366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:01.731366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:29:01.731366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:01.731366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:01.731366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.731366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:01.731769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:01.731769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:01.731769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.731769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:01.731769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:01.731769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:01.731769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:01.731769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:01.731769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:01.731769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:01.731769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:01.731769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.731769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:01.731769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:01.732045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.732045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:01.732045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.732045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:01.734051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.734051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:01.734051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:01.734051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:01.734051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:01.734051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:01.734051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.734051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:01.734051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:01.734051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.734051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:01.734051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:01.734051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.734051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:01.734051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:01.734051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:01.735160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:01.735610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.735610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:01.735610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:01.735610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.735610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:01.735610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.735610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:01.735610 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:01.735610 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:01.735878 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:01.735878 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:01.736397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:01.736731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.737397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:01.737397 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.737397 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:01.737397 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:01.737397 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.737397 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:01.737397 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.737397 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:01.738509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.738509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:01.738509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:01.738509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:01.738509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.738509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:01.738509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:01.738509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:01.738509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.738509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:01.738509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:01.738509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:01.738509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:01.738509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:01.738509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:01.739084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:01.739084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:01.739084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.739084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:01.739084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:01.739084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.739084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:01.739084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.739084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:01.739431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.739431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:01.739431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:01.739431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:01.739431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.739431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:01.739431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:01.739431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:01.739431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:01.739431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.739431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:01.739796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:01.739796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:01.739796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.739796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:01.739796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:01.739796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:01.739796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:01.739796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:01.739796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:01.739796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:01.739796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:01.739796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.739796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:01.739796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:01.740300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.740300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:01.740300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.740300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.740617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:01.741057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:01.741452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.741452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:01.741452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.741452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:01.743844 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:01.744291 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:01.744972 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:01.745207 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:01.745836 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:01.746146 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:01.746465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.747149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:01.747149 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.747149 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:01.747149 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:01.747149 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.747149 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:01.747149 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.747149 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:01.747912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.747912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:01.747912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:01.747912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:01.747912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.747912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:01.747912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:01.747912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.747912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:01.747912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:01.747912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:01.747912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.748431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:01.748431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:01.748431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:01.748431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:01.748431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:01.748431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:01.748431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:01.748431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:01.748431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.748431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:01.748431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:01.748431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.748431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:01.748727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.748727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:01.748727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.748727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:01.748727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:01.748727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:01.748727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:01.748727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:01.748727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.748727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:01.748727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:01.748727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.748727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:01.748727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:01.748727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:01.748969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.749253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:01.749253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:01.749253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.749253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:01.749253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.749253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:01.839460 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:01.844470 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:01.844470 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:01.845247 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:01.919522 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:01.919522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.922841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:01.922841 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.922841 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:01.922841 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:01.922841 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.922841 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:01.922841 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.922841 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:01.924583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.078 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.924583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:01.924583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:01.924583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:01.924583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.924583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:01.924583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:01.924583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.924583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:01.924583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:01.924583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:01.924583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.924986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:01.924986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:01.924986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:01.924986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:01.924986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:01.924986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:01.924986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:01.924986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:01.924986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.924986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:01.924986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:01.924986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.924986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:01.925290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.925290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:01.925616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.925616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:01.925616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:01.925616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:01.925616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:01.925616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:01.925616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.925616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:01.925616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:01.925616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.925616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:01.925616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:01.925616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.925616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:01.925616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:01.925616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:01.926430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:01.927741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:01.927741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:01.927741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:06.707966 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:06.710136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:06.710136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:06.711167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:06.711504 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:06.711757 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:06.711991 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:06.712089 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:06.712261 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:06.712311 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:06.712966 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:06.712966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.715818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:06.715818 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.715818 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:06.715818 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:06.715818 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.715818 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:06.715818 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.715818 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:06.716559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.716559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:06.716559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:06.716559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:06.716559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.716559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:06.716559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:06.716559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.716559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:06.716559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:06.716559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:06.716559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.717405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:06.717405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:06.717405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:06.717405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:06.717405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:06.717405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:06.717405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:06.717405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:06.717405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.717405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:06.717405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:06.717405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.717405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:06.717934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.717934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:06.719786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.719786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:06.719786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:06.719786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:06.719786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:06.719786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:06.719786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.719786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:06.719786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:06.719786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.719786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:06.719786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:06.719786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.719786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:06.719786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:06.719786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:06.720253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.720821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:06.720821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.720821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:06.722762 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:06.723510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:06.723700 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:06.723803 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:29:06.725286 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:06.725286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.725613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:06.725613 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.725613 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:06.725613 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:06.725613 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.725613 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:06.725613 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.725613 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:06.726258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.726258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:06.726258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:06.726258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:06.726258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.726258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:06.726258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:06.726258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:06.726258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.726258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:06.726258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:06.726258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:06.726258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:06.726258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:06.726258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:06.726827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:06.726827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:06.726827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.726827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:06.726827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:06.726827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.726827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:06.726827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.726827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:06.727108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.727108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:06.727108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:06.727108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:06.727108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.727108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:06.727108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:06.727108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:29:06.727108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:06.727108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:06.727108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.727108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:06.727474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:06.727474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:06.727474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.727474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:06.727474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:06.727474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:06.727474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:06.727474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:06.727474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:06.727474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:06.727474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:06.727474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.727474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:06.727474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:06.728752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.728752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:06.728752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.728752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:06.730022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.730022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:06.730022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:06.730022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:06.730022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:06.730022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:06.730022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.730022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:06.730022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:06.730022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.730022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:06.730022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:06.730022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.730022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:06.730022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:06.730022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:06.730570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:06.731141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.731141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:06.731141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:06.731141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.731141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:06.731141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.731141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:06.736824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:06.738171 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:06.738675 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:06.738675 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:06.739030 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:06.739303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.740759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:06.740759 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.740759 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:06.740759 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:06.740759 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.740759 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:06.740759 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.740759 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:06.743597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.743597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:06.743597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:06.743597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:06.743597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.743597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:06.743597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:06.743597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:06.743597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.743597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:06.743597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:06.743597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:06.743597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:06.743597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:06.743597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:06.744141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:06.744141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:06.744141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.744141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:06.744141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:06.744141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.744141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:06.744141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.744141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:06.745630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.745630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:06.745630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:06.745630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:06.745630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.745630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:06.745630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:06.745630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:06.745630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:06.745630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.745630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:06.746210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:06.746210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:06.746210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.746210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:06.746210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:06.746210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:06.746210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:06.746210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:06.746210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:06.746210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:06.746210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:06.746210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.746210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:06.746210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:06.746587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.746587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:06.746587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.746587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:06.746587 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.746960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:06.747424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:06.747953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.747953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:06.747953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.747953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:06.747953 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:06.747953 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:06.747953 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:06.748161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:06.748381 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:06.748773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.749234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:06.749234 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.749234 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:06.749234 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:06.749234 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.749234 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:06.749234 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.749234 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:06.750280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.750280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:06.750280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:06.750280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:06.750280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.750280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:06.750280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:06.750280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.750280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:06.750280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:06.750280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:06.750280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.750703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:06.750703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:06.750703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:06.750703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:06.750703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:06.750703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:06.750703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:06.750703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:06.750703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.750703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:06.750703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:06.750703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.750703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:06.751095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.751095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:06.751705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.751705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:06.751705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:06.751705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:06.751705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:06.751705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:06.751705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.751705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:06.751705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:06.751705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.751705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:06.751705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:06.751705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.751705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:06.751705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:06.751705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:06.752196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.752795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:06.752795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.752795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:06.844738 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:06.845910 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:06.846075 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:06.847093 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:06.914952 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:06.914952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.915363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:06.915363 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.915363 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:06.915363 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:06.915363 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.915363 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:06.915363 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.915363 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:06.916419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.069 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.916419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:06.916419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:06.916419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:06.916419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.916419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:06.916419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:06.916419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.916419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:06.916419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:06.916419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:06.916419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.916849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:06.916849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:06.916849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:06.916849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:06.916849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:06.916849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:06.916849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:06.916849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:06.916849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.916849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:06.916849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:06.916849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.916849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:06.917147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.917147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:06.917881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.917881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:06.917881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:06.917881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:06.917881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:06.917881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:06.917881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.917881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:06.917881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:06.917881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.917881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:06.917881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:06.917881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.917881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:06.917881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:06.917881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:06.918388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:06.918898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:06.918898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:06.918898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:11.710327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:11.711991 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:11.712117 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:11.713864 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:11.713864 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:11.714202 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:11.714202 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:11.714367 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:11.714561 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:11.714744 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:11.714904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:11.715371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.716912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:11.716912 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.716912 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:11.716912 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:11.716912 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.716912 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:11.716912 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.716912 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:11.717386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.717386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:11.717386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:11.717386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:11.717386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.717386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:11.717386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:11.717386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.717386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:11.717386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:11.717386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:11.717386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.717827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:11.717827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:11.717827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:11.717827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:11.717827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:11.717827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:11.717827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:11.717827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:11.717827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.717827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:11.717827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:11.717827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.717827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:11.718096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.718096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:11.718442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.718442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:11.718442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:11.718442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:11.718442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:11.718442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:11.718442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.718442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:11.718442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:11.718442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.718442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:11.718442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:11.718442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.718442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:11.718442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:11.718442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:11.719180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.719758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:11.719758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.719758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:11.724160 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:11.724160 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:11.724611 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:11.724611 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:29:11.724990 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:11.725237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.726136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:11.726136 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.726136 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:11.726136 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:11.726136 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.726136 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:11.726136 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.726136 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:11.727224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.727224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:11.727224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:11.727224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:11.727224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.727224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:11.727224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:11.727224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:11.727224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.727224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:11.727224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:11.727224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:11.727224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:11.727224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:11.727224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:11.727896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:11.727896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:11.727896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.727896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:11.727896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:11.727896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.727896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:11.727896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.727896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:11.728326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.728326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:11.728326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:11.728326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:11.728326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.728326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:11.728326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:11.728326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:29:11.728326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:11.728326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:11.728326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.728326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:11.728782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:11.728782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:11.728782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.728782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:11.728782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:11.728782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:11.728782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:11.728782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:11.728782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:11.728782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:11.728782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:11.728782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.728782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:11.728782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:11.729166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.729166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:11.729166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.729166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:11.729286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.729286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:11.729286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:11.729286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:11.729286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:11.729286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:11.729286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.729286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:11.729286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:11.729286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.729286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:11.729286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:11.729286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.729286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:11.729286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:11.729286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:11.729803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:11.730916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.730916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:11.730916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:11.730916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.730916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:11.730916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.730916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:11.738400 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:11.738965 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:11.739488 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:11.739488 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:11.740089 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:11.740456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.741969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:11.741969 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.741969 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:11.741969 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:11.741969 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.741969 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:11.741969 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.741969 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:11.742532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.742532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:11.742532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:11.742532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:11.742532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.742532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:11.742532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:11.742532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:11.742532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.742532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:11.742532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:11.742532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:11.742532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:11.742532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:11.742532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:11.743254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:11.743254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:11.743254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.743254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:11.743254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:11.743254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.743254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:11.743254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.743254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:11.743633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.743633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:11.743633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:11.743633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:11.743633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.743633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:11.743633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:11.743633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:11.743633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:11.743633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.743633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:11.744023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:11.744023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:11.744023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.744023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:11.744023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:11.744023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:11.744023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:11.744023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:11.744023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:11.744023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:11.744023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:11.744023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.744023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:11.744023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:11.744279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.744279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:11.744279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.744279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.744837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:11.745209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:11.745607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.745607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:11.745607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.745607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:11.747363 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:11.748005 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:11.748005 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:11.748194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:11.748459 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:11.748910 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:11.749209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.749773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:11.749773 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.749773 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:11.749773 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:11.749773 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.749773 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:11.749773 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.749773 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:11.750555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.750555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:11.750555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:11.750555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:11.750555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.750555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:11.750555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:11.750555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.750555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:11.750555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:11.750555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:11.750555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.751014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:11.751014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:11.751014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:11.751014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:11.751014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:11.751014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:11.751014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:11.751014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:11.751014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.751014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:11.751014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:11.751014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.751014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:11.751433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.751433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:11.751697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.751697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:11.751697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:11.751697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:11.751697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:11.751697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:11.751697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.751697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:11.751697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:11.751697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.751697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:11.751697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:11.751697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.751697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:11.751697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:11.751697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:11.752178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.752752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:11.752752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.752752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:11.846473 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:11.846980 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:11.847161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:11.847919 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:11.897496 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:11.897708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.898151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:11.898151 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.898151 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:11.898151 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:11.898151 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.898151 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:11.898151 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.898151 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:11.898901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.051 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.898901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:11.898901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:11.898901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:11.898901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.898901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:11.898901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:11.898901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.898901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:11.898901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:11.898901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:11.898901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.899282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:11.899282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:11.899282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:11.899282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:11.899282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:11.899282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:11.899282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:11.899282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:11.899282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.899282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:11.899282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:11.899282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.899282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:11.900121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.900121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:11.900121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.900121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:11.900121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:11.900121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:11.900121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:11.900121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:11.900121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.900121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:11.900121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:11.900121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.900121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:11.900121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:11.900121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:11.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:11.902933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:11.902933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:11.902933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:11.902933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:11.902933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:11.902933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:16.711984 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:16.713040 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:16.713040 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:16.713835 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:16.713968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:16.714219 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:16.714320 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:16.714392 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:16.715064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:16.715064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:16.715064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:16.715064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.715391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:16.715391 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.715391 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:16.715391 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:16.715391 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.715391 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:16.715391 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.715391 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:16.716426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.716426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:16.716426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:16.716426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:16.716426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.716426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:16.716426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:16.716426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.716426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:16.716426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:16.716426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:16.716426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.717663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:16.717663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:16.717663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:16.717663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:16.717663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:16.717663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:16.717663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:16.717663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:16.717663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.717663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:16.717663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:16.717663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.717663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:16.718000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.718000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:16.718000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.718000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:16.718000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:16.718000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:16.718000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:16.718000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:16.718000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.718000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:16.718000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:16.718000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.718000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:16.718000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:16.718000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:16.718363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.718852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:16.718852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:16.718852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.718852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:16.718852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.718852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:16.724167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:16.725388 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:16.725388 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:16.725679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:29:16.726177 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:16.726177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.726516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:16.726516 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.726516 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:16.726516 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:16.726516 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.726516 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:16.726516 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.726516 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:16.727420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.727420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:16.727420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:16.727420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:16.727420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.727420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:16.727420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:16.727420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:16.727420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.727420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:16.727420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:16.727420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:16.727420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:16.727420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:16.727420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:16.727964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:16.727964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:16.727964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.727964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:16.727964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:16.727964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.727964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:16.727964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.727964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:16.728228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.728228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:16.728228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:16.728228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:16.728228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.728228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:16.728228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:16.728228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:29:16.728228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:16.728228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:16.728228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.728228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:16.728588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:16.728588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:16.728588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.728588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:16.728588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:16.728588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:16.728588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:16.728588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:16.728588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:16.728588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:16.728588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:16.728588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.728588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:16.728588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:16.729056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.729056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:16.729056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.729056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:16.729177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.729177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:16.729177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:16.729177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:16.729177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:16.729177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:16.729177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.729177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:16.729177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:16.729177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.729177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:16.729177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:16.729177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.729177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:16.729177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:16.729177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:16.729567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:16.730184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.730184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:16.730184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:16.730184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.730184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:16.730184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.730184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:16.739122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:16.740950 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:16.741587 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:16.741587 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:16.742046 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:16.742416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.743330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:16.743330 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.743330 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:16.743330 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:16.743330 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.743330 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:16.743330 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.743330 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:16.746248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.746248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:16.746248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:16.746248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:16.746248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.746248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:16.746248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:16.746248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:16.746248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.746248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:16.746248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:16.746248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:16.746248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:16.746248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:16.746248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:16.747001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:16.747001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:16.747001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.747001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:16.747001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:16.747001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.747001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:16.747001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.747001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:16.747001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.747001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:16.747001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:16.747001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:16.747001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.747445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:16.747445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:16.747445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:16.747445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:16.747445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.747445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:16.747445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:16.747445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:16.747445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.747445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:16.747445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:16.747445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:16.747445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:16.748118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:16.748118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:16.748118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:16.748118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:16.748118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.748118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:16.748118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:16.748118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.748118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:16.748118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.748118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:16.748118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:16.748398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.748888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:16.749328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:16.749328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:16.749328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:16.749328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:16.751689 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:16.751689 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:16.751885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.752408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:16.752408 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.752408 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:16.752408 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:16.752408 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.752408 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:16.752408 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.752408 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:16.753302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.753302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:16.753302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:16.753302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:16.753302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.753302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:16.753302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:16.753302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.753302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:16.753302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:16.753302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:16.753302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.753724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:16.753724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:16.753724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:16.753724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:16.753724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:16.753724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:16.753724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:16.753724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:16.753724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.753724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:16.753724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:16.753724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.753724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:16.754002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.754002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:16.754633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.754633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:16.754633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:16.754633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:16.754633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:16.754633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:16.754633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.754633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:16.754633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:16.754633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.754633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:16.754633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:16.754633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.754633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:16.754633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:16.754633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:16.755101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.755580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:16.755580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.755580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:16.847427 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:16.848798 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:16.848952 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:16.850436 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:16.926886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:16.926886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.931208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:16.931208 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.931208 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:16.931208 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:16.931208 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.931208 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:16.931208 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.931208 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:16.933690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.082 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.933690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:16.933690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:16.933690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:16.933690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.933690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:16.933690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:16.933690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.933690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:16.933690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:16.933690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:16.933690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.934190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:16.934190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:16.934190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:16.934190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:16.934190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:16.934190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:16.934190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:16.934190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:16.934190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.934190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:16.934190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:16.934190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.934190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:16.934570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.934570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:16.934570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.934570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:16.934570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:16.934570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:16.934570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:16.934570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:16.934570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.934570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:16.934570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:16.934570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.934570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:16.934570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:16.934570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:16.935036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:16.935712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:16.935712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:16.935712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:16.935712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:16.935712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:16.935712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:21.716775 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:21.717419 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:21.717419 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:21.719879 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:21.720066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:21.720293 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:21.720373 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:21.720466 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:21.720594 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:21.720787 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-31445a58-14ab-4255-83dd-8d63b7030d94, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:21.720915 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:21.721064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.721605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:21.721605 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.721605 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:21.721605 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:21.721605 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.721605 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:21.721605 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.721605 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:21.722429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.722429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:21.722429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:21.722429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:21.722429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.722429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:21.722429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:21.722429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.722429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:21.722429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:21.722429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:21.722429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.722780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:21.722780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:21.722780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:21.722780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:21.722780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:21.722780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:21.722780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:21.722780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:21.722780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.722780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:21.722780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:21.722780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.722780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:21.723079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.723079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:21.723773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.723773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:21.723773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:21.723773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:21.723773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:21.723773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:21.723773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.723773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:21.723773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:21.723773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.723773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:21.723773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:21.723773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.723773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:21.723773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:21.723773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:21.724356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.724956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:21.724956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.724956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:21.726818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:21.727093 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:21.727202 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:21.727314 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:29:21.727643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:21.727896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.728362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:21.728362 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.728362 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:21.728362 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:21.728362 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.728362 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:21.728362 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.728362 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:21.729065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.729065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:21.729065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:21.729065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:21.729065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.729065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:21.729065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:21.729065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:21.729065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.729065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:21.729065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:21.729065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:21.729065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:21.729065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:21.729065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:21.730169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:21.730169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:21.730169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.730169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:21.730169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:21.730169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.730169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:21.730169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.730169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:21.730169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.730169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:21.730169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:21.730169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:21.730169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.730737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:21.730737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:21.730737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:29:21.730737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:21.730737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:21.730737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.730737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:21.730737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:21.730737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:21.730737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.730737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:21.730737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:21.730737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:21.731083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:21.731083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:21.731083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:21.731083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:21.731083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:21.731083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.731083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:21.731083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:21.731083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.731083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:21.731083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.731083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:21.733169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.733169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:21.733169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:21.733169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:21.733169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:21.733169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:21.733169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.733169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:21.733169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:21.733169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.733169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:21.733169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:21.733169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.733169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:21.733169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:21.733169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:21.733797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:21.734369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.734369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:21.734369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:21.734369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.734369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:21.734369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.734369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:21.741479 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:21.741479 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:21.741479 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:21.741479 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:21.743333 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:21.744204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.745470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:21.745470 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.745470 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:21.745470 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:21.745470 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.745470 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:21.745470 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.745470 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:21.747300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.747300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:21.747300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:21.747300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:21.747300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.747300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:21.747300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:21.747300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:21.747300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.747300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:21.747300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:21.747300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:21.747300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:21.747300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:21.747300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:21.748616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:21.748616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:21.748616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.748616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:21.748616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:21.748616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.748616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:21.748616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.748616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:21.756020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.756020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:21.756020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:21.756020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:21.756020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.756020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:21.756020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:21.756020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:21.756020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:21.756020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.756020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:21.756465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:21.756465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:21.756465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.756465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:21.756465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:21.756465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:21.756465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:21.756465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:21.756465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:21.756465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:21.756465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:21.756465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.756465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:21.756465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:21.756888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.756888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:21.756888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.756888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.761594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:21.762204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:21.762749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.762749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:21.762749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.762749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:21.763142 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:21.764327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:21.765802 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:21.766113 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:21.766588 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:21.766817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:21.767006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.767412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:21.767412 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.767412 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:21.767412 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:21.767412 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.767412 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:21.767412 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.767412 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:21.768237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.768237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:21.768237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:21.768237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:21.768237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.768237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:21.768237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:21.768237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.768237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:21.768237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:21.768237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:21.768237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.768729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:21.768729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:21.768729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:21.768729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:21.768729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:21.768729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:21.768729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:21.768729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:21.768729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.768729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:21.768729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:21.768729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.768729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:21.770207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.770207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:21.770207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.770207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:21.770207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:21.770207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:21.770207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:21.770207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:21.770207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.770207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:21.770207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:21.770207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.770207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:21.770207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:21.770207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:21.771186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.771495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:21.771495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:21.771495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.771495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:21.771495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.771495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:21.851589 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:21.851589 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:21.851589 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:21.851589 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:21.869108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=1824,neutron:UPDATE=5,neutron:DELETE=7 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:29:21.973295 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:21.973295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.973295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:21.973295 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.973295 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:21.973295 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:21.973295 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.973295 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:21.973295 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.973295 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:21.974132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.123 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.974132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:21.974132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:21.974132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:21.974132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.974132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:21.974132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:21.974132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.974132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:21.974132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:21.974132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:21.974132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.975838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:21.975838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:21.975838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:21.975838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:21.975838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:21.975838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:21.975838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:21.975838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:21.975838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.975838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:21.975838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:21.975838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.975838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:21.976571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.976571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:21.976571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.976571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:21.976571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:21.976571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:21.976571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:21.976571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:21.976571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.976571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:21.976571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:21.976571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.976571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:21.976571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:21.976571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:21.977146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:21.978262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:21.978262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:21.978262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:21.978262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:21.978262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:21.978262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:26.717243 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:26.718119 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:26.718249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:26.719465 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:26.719805 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:26.720010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:26.720143 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:26.720263 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:26.720391 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:26.720513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:26.720712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.721914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:26.721914 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.721914 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:26.721914 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:26.721914 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.721914 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:26.721914 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.721914 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:26.722230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.722230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:26.722230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:26.722230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:26.722230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.722230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:26.722230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:26.722230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.722230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:26.722230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:26.722230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:26.722230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.722671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:26.722671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:26.722671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:26.722671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:26.722671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:26.722671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:26.722671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:26.722671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:26.722671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.722671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:26.722671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:26.722671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.722671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:26.723071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.723071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:26.723071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.723071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:26.723071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:26.723071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:26.723071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:26.723071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:26.723071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.723071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:26.723071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:26.723071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.723071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:26.723071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:26.723071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:26.723505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.724118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:26.724118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:26.724118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.724118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:26.724118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.724118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:26.732780 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:26.732780 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:26.732780 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:26.732780 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:29:26.732780 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:26.732780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.733256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:26.734699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:26.734699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:26.734699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:26.734699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:26.734699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:26.734699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:26.734699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:26.734699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.734699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:26.734699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:26.734699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.734699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:26.734699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.734699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:26.734699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.735234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:26.735234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:26.735234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:26.735234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.735234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:26.735234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:26.735234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:29:26.735234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:26.735234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:26.735234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.735234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:26.735234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:26.735234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:26.735764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.735764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:26.735764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:26.735764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:26.735764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:26.735764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:26.735764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:26.735764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:26.735764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:26.735764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.735764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:26.735764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:26.735764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.735764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:26.736371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.736371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:26.736371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.736371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:26.736371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:26.736371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:26.736371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:26.736371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:26.736371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.736371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:26.736371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:26.736371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.736371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:26.736371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:26.736371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:26.737298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:26.737935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:26.737935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:26.737935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.737935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:26.737935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:26.737935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.737935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:26.737935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.737935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:26.743887 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:26.743887 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:26.743887 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:26.743887 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:26.745465 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:26.745842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.747459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:26.747459 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.747459 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:26.747459 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:26.747459 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.747459 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:26.747459 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.747459 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:26.749339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.749339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:26.749339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:26.749339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:26.749339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.749339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:26.749339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:26.749339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:26.749339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.749339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:26.749339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:26.749339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:26.749339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:26.749339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:26.749339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:26.750370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:26.750370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:26.750370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.750370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:26.750370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:26.750370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.750370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:26.750370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.750370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:26.750778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.750778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:26.750778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:26.750778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:26.750778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.750778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:26.750778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:26.750778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:26.750778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:26.750778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.750778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:26.751122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:26.751122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:26.751122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.751122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:26.751122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:26.751122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:26.751122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:26.751122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:26.751122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:26.751122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:26.751122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:26.751122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.751122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:26.751122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:26.751620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.751620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:26.751620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.751620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.752078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:26.752625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:26.753196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.753196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:26.753196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.753196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:26.764668 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:26.765032 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:26.765244 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:26.765320 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:26.767752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:26.768111 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:26.768477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.769304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:26.769304 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.769304 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:26.769304 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:26.769304 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.769304 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:26.769304 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.769304 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:26.771384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.771384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:26.771384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:26.771384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:26.771384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.771384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:26.771384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:26.771384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.771384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:26.771384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:26.771384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:26.771384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.772377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:26.772377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:26.772377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:26.772377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:26.772377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:26.772377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:26.772377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:26.772377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:26.772377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.772377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:26.772377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:26.772377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.772377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:26.772807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.772807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:26.775406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.775406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:26.775406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:26.775406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:26.775406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:26.775406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:26.775406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.775406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:26.775406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:26.775406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.775406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:26.775406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:26.775406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.775406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:26.775406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:26.775406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:26.776031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.776648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:26.776648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.776648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:26.854261 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:26.854830 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:26.854947 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:26.855972 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:26.954024 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:26.954726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.956061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:26.956061 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.956061 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:26.956061 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:26.956061 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.956061 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:26.956061 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.956061 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:26.958527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.102 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.958527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:26.958527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:26.958527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:26.958527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.958527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:26.958527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:26.958527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.958527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:26.958527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:26.958527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:26.958527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.959051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:26.959051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:26.959051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:26.959051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:26.959051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:26.959051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:26.959051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:26.959051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:26.959051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.959051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:26.959051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:26.959051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.959051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:26.959399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.959399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:26.960678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.960678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:26.960678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:26.960678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:26.960678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:26.960678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:26.960678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.960678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:26.960678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:26.960678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.960678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:26.960678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:26.960678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.960678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:26.960678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:26.960678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:26.961096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:26.961457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:26.961457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:26.961457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:31.718358 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:31.719164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:31.719164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:31.721865 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:31.722596 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:31.722821 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:31.723257 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:31.724308 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:31.724308 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f7bee114-f44a-462c-b1b8-38a0e8f84bb1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:31.728767 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:31.729363 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:31.730052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.732428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:31.732428 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.732428 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:31.732428 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:31.732428 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.732428 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:31.732428 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.732428 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:31.734470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.013 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.734470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:31.734470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:31.734470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:31.734470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.734470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:31.734470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:31.734470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.734470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:31.734470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:31.734470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:31.734470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.735934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:31.735934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:31.735934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:31.735934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:31.735934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:31.735934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:31.735934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:31.735934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:31.735934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.735934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:31.735934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:31.735934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.735934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:31.736718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.736718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:31.736718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.736718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:31.736718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:31.736718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:31.736718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:31.736718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:31.736718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.736718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:31.736718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:31.736718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.736718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:31.736718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:31.736718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:31.737160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.737704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:31.737704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:31.737704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.737704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:31.737704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.737704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:31.737704 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:31.737704 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:31.737704 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:29:31.737704 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:31.737992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.737992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:31.737992 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.737992 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:31.737992 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:31.737992 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.737992 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:31.737992 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.737992 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:31.739287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.739287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:31.739287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:31.739287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:31.739287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.739287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:31.739287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:31.739287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:31.739287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.739287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:31.739287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:31.739287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:31.739287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:31.739287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:31.739287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:31.739838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:31.739838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:31.739838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.739838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:31.739838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:31.739838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.739838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:31.739838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.739838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:31.739838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.739838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:31.739838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:31.739838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:31.739838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.740275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:31.740275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:31.740275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:29:31.740275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:31.740275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:31.740275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.740275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:31.740275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:31.740275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:31.740275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.740275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:31.740275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:31.740275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:31.740591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:31.740591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:31.740591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:31.740591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:31.740591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:31.740591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.740591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:31.740591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:31.740591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.740591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:31.740591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.740591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:31.741134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.741134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:31.741134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:31.741134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:31.741134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:31.741134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:31.741134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.741134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:31.741134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:31.741134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.741134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:31.741134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:31.741134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.741134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:31.741134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:31.741134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:31.741477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:31.742035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.742035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:31.742035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:31.742035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.742035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:31.742035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.742035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:31.742522 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:31.743683 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:31.743683 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:31.743683 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:31.749881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:31.750189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.751015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:31.751015 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.751015 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:31.751015 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:31.751015 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.751015 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:31.751015 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.751015 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:31.752932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.752932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:31.752932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:31.752932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:31.752932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.752932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:31.752932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:31.752932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:31.752932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.752932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:31.752932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:31.752932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:31.752932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:31.752932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:31.752932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:31.753394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:31.753394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:31.753394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.753394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:31.753394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:31.753394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.753394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:31.753394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.753394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:31.754507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.754507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:31.754507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:31.754507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:31.754507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.754507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:31.754507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:31.754507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:31.754507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:31.754507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.754507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:31.754919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:31.754919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:31.754919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.754919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:31.754919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:31.754919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:31.754919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:31.754919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:31.754919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:31.754919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:31.754919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:31.754919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.754919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:31.754919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:31.755624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.755624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:31.755624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.755624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:31.755624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.755624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:31.755624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:31.755624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:31.755624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:31.755624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:31.755624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.755624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:31.755624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:31.755624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:31.756302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:31.757041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:31.757041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:31.757041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:31.757041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:31.757041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.757041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:31.757041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:31.757041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.757041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:31.757041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.757041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:31.765179 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:31.765577 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:31.765767 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:31.766021 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:31.767049 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:31.767375 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:31.767716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.768487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:31.768487 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.768487 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:31.768487 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:31.768487 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.768487 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:31.768487 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.768487 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:31.769519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.769519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:31.769519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:31.769519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:31.769519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.769519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:31.769519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:31.769519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.769519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:31.769519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:31.769519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:31.769519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.769947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:31.769947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:31.769947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:31.769947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:31.769947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:31.769947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:31.769947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:31.769947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:31.769947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.769947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:31.769947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:31.769947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.769947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:31.770280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.770280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:31.770584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.770584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:31.770584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:31.770584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:31.770584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:31.770584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:31.770584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.770584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:31.770584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:31.770584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.770584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:31.770584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:31.770584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.770584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:31.770584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:31.770584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:31.772144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.772769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:31.772769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.772769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:31.854867 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:31.855416 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:31.855553 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:31.856495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:31.920708 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:31.921274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.922023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:31.922023 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.922023 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:31.922023 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:31.922023 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.922023 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:31.922023 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.922023 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:31.923268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.066 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.923268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:31.923268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:31.923268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:31.923268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.923268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:31.923268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:31.923268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.923268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:31.923268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:31.923268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:31.923268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.925491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:31.925491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:31.925491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:31.925491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:31.925491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:31.925491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:31.925491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:31.925491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:31.925491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.925491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:31.925491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:31.925491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.925491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:31.925807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.925807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:31.925807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.925807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:31.925807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:31.925807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:31.925807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:31.925807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:31.925807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.925807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:31.925807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:31.925807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.925807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:31.925807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:31.925807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:31.926096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:31.926560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:31.926560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:31.926560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:31.926560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:31.926560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:31.926560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:36.723014 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:36.723695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:36.723695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:36.728723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:36.728723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:36.729287 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:36.729719 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:36.730028 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:36.730275 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-21537149-b4d5-4fe6-9a59-2cd7a59048e4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:36.730557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.735210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:36.735210 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.735210 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:36.735210 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:36.735210 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.735210 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:36.735210 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.735210 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:36.736753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.012 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.736753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:36.736753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:36.736753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:36.736753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.736753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:36.736753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:36.736753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.736753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:36.736753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:36.736753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:36.736753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.737149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:36.737149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:36.737149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:36.737149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:36.737149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:36.737149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:36.737149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:36.737149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:36.737149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.737149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:36.737149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:36.737149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.737149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:36.737451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.737451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:36.737519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.737519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:36.737519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:36.737519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:36.737519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:36.737519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:36.737519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.737519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:36.737519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:36.737519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.737519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:36.737519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:36.737519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.737519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:36.737519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:36.737519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:36.738196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.738695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:36.738695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.738695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:36.747423 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:36.747423 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:36.750785 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:36.751075 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:36.751260 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:29:36.753412 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:36.753412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.753755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:36.753755 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.753755 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:36.753755 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:36.753755 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.753755 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:36.753755 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.753755 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:36.754366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.754366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:36.754366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:36.754366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:36.754366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.754366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:36.754366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:36.754366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:36.754366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.754366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:36.754366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:36.754366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:36.754366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:36.754366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:36.754366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:36.754820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:36.754820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:36.754820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.754820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:36.754820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:36.754820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.754820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:36.754820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.754820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:36.755241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.755241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:36.755241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:36.755241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:36.755241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.755241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:36.755241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:36.755241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:29:36.755241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:36.755241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:36.755241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.755241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:36.755735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:36.755735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:36.755735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.755735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:36.755735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:36.755735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:36.755735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:36.755735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:36.755735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:36.755735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:36.755735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:36.755735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.755735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:36.755735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:36.756063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.756063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:36.756063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.756063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:36.756063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.756063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:36.756063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:36.756063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:36.756063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:36.756063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:36.756063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.756063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:36.756063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:36.756063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:36.757003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:36.758170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:36.758170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:36.758170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:36.758170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:36.758170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:36.758170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.758170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:36.758170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:36.758170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.758170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:36.758170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.758170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:36.762724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:36.762724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:36.762724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:36.762724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:36.762724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.765526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:36.765526 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.765526 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:36.765526 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:36.765526 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.765526 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:36.765526 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.765526 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:36.766678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.766678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:36.766678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:36.766678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:36.766678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.766678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:36.766678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:36.766678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:36.766678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.766678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:36.766678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:36.766678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:36.766678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:36.766678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:36.766678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:36.767132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:36.767132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:36.767132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.767132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:36.767132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:36.767132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.767132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:36.767132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.767132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:36.769318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.769318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:36.769318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:36.769318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:36.769318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.769318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:36.769318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:36.769318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:36.769318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:36.769318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.769318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:36.769857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:36.769857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:36.769857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.769857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:36.769857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:36.769857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:36.769857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:36.769857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:36.769857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:36.769857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:36.769857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:36.769857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.769857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:36.769857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:36.770286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.770286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:36.770286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.770286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:36.770286 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.771762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:36.773726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:36.774921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.774921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:36.774921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.774921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:36.774921 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:36.774921 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:36.774921 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:36.775264 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:36.775986 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:36.776911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.778866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:36.778866 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.778866 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:36.778866 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:36.778866 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.778866 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:36.778866 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.778866 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:36.779398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.779398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:36.779398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:36.779398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:36.779398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.779398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:36.779398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:36.779398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.779398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:36.779398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:36.779398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:36.779398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.779818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:36.779818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:36.779818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:36.779818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:36.779818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:36.779818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:36.779818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:36.779818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:36.779818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.779818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:36.779818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:36.779818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.779818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:36.780196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.780196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:36.782392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.782392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:36.782392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:36.782392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:36.782392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:36.782392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:36.782392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.782392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:36.782392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:36.782392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.782392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:36.782392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:36.782392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.782392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:36.782392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:36.782392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:36.783032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.783713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:36.783713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.783713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:36.857578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:36.857578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:36.857578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:36.859818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:36.937789 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:36.938037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.940331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:36.940331 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.940331 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:36.940331 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:36.940331 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.940331 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:36.940331 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.940331 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:36.941588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.083 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.941588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:36.941588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:36.941588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:36.941588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.941588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:36.941588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:36.941588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.941588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:36.941588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:36.941588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:36.941588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.942017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:36.942017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:36.942017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:36.942017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:36.942017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:36.942017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:36.942017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:36.942017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:36.942017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.942017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:36.942017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:36.942017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.942017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:36.942327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.942327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:36.944335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.944335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:36.944335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:36.944335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:36.944335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:36.944335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:36.944335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.944335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:36.944335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:36.944335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.944335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:36.944335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:36.944335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.944335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:36.944335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:36.944335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:36.944876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:36.945326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:36.945326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:36.945326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:41.723512 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:41.724043 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:41.724154 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:41.725796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:41.727087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:41.727768 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:41.728709 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:41.728990 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:41.729620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.732561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:41.732561 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.732561 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:41.732561 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:41.732561 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.732561 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:41.732561 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.732561 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:41.735429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.735429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:41.735429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:41.735429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:41.735429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.735429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:41.735429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:41.735429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.735429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:41.735429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:41.735429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:41.735429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.735861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:41.735861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:41.735861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:41.735861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:41.735861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:41.735861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:41.735861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:41.735861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:41.735861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.735861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:41.735861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:41.735861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.735861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:41.737571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.737571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:41.738272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.738272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:41.738272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:41.738272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:41.738272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:41.738272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:41.738272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.738272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:41.738272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:41.738272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.738272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:41.738272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:41.738272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.738272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:41.738272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:41.738272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:41.738884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.739449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:41.739449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.739449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:41.751072 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:41.751359 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:41.751583 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:41.751776 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:29:41.752823 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:41.752823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.753654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:41.753654 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.753654 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:41.753654 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:41.753654 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.753654 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:41.753654 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.753654 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:41.754719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.754719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:41.754719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:41.754719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:41.754719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.754719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:41.754719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:41.754719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:41.754719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.754719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:41.754719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:41.754719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:41.754719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:41.754719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:41.754719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:41.755608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:41.755608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:41.755608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.755608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:41.755608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:41.755608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.755608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:41.755608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.755608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:41.755608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.755608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:41.755608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:41.755608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:41.755608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.757009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:41.757009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:41.757009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:29:41.757009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:41.757009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:41.757009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.757009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:41.757009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:41.757009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:41.757009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.757009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:41.757009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:41.757009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:41.757363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:41.757363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:41.757363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:41.757363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:41.757363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:41.757363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.757363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:41.757363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:41.757363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.757363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:41.757363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.757363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:41.757701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.757701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:41.757701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:41.757701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:41.757701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:41.757701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:41.757701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.757701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:41.757701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:41.757701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.757701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:41.757701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:41.757701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.757701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:41.757701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:41.757701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:41.758109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:41.758589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.758589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:41.758589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:41.758589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.758589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:41.758589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.758589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:41.759245 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:41.760097 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:41.760309 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:41.760564 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:41.762308 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:41.762308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.762576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:41.762576 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.762576 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:41.762576 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:41.762576 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.762576 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:41.762576 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.762576 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:41.763649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.763649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:41.763649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:41.763649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:41.763649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.763649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:41.763649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:41.763649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:41.763649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.763649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:41.763649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:41.763649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:41.763649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:41.763649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:41.763649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:41.764107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:41.764107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:41.764107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.764107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:41.764107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:41.764107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.764107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:41.764107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.764107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:41.766903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.766903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:41.766903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:41.766903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:41.766903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.766903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:41.766903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:41.766903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:41.766903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:41.766903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.766903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:41.767419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:41.767419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:41.767419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.767419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:41.767419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:41.767419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:41.767419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:41.767419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:41.767419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:41.767419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:41.767419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:41.767419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.767419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:41.767419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:41.767800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.767800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:41.767800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.767800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:41.767800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.767800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:41.767800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:41.767800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:41.767800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:41.767800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:41.767800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.767800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:41.767800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:41.767800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:41.768308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:41.770133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:41.770133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:41.770133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:41.770133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:41.770133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.770133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:41.770133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:41.770133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.770133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:41.770133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.770133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:41.773069 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:41.774919 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:41.775083 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:41.775322 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:41.776690 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:41.776690 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:41.776690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.778115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:41.778115 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.778115 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:41.778115 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:41.778115 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.778115 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:41.778115 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.778115 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:41.778387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.778387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:41.778387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:41.778387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:41.778387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.778387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:41.778387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:41.778387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.778387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:41.778387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:41.778387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:41.778387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.778742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:41.778742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:41.778742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:41.778742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:41.778742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:41.778742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:41.778742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:41.778742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:41.778742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.778742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:41.778742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:41.778742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.778742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:41.779061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.779061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:41.779302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.779302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:41.779302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:41.779302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:41.779302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:41.779302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:41.779302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.779302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:41.779302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:41.779302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.779302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:41.779302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:41.779302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.779302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:41.779302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:41.779302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:41.779970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.780530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:41.780530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.780530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:41.857439 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:41.858314 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:41.858314 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:41.858930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:41.925657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:41.925906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.926517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:41.926517 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.926517 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:41.926517 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:41.926517 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.926517 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:41.926517 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.926517 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:41.927890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.068 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.927890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:41.927890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:41.927890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:41.927890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.927890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:41.927890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:41.927890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.927890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:41.927890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:41.927890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:41.927890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.928215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:41.928215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:41.928215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:41.928215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:41.928215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:41.928215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:41.928215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:41.928215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:41.928215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.928215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:41.928215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:41.928215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.928215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:41.928583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.928583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:41.929033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.929033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:41.929033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:41.929033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:41.929033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:41.929033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:41.929033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.929033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:41.929033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:41.929033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.929033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:41.929033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:41.929033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.929033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:41.929033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:41.929033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:41.929493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:41.930032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:41.930032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:41.930032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:46.724498 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:46.725092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:46.725142 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:46.726710 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:46.728869 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:46.731402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:46.732215 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:46.733971 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:46.734127 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:46.734590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.740595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:46.740595 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.740595 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:46.740595 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:46.740595 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.740595 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:46.740595 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.740595 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:46.742060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.015 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.742060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:46.742060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:46.742060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:46.742060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.742060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:46.742060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:46.742060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.742060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:46.742060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:46.742060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:46.742060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.742395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:46.742395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:46.742395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:46.742395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:46.742395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:46.742395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:46.742395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:46.742395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:46.742395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.742395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:46.742395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:46.742395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.742395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:46.742711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.742711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:46.743012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.743012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:46.743012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:46.743012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:46.743012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:46.743012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:46.743012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.743012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:46.743012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:46.743012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.743012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:46.743012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:46.743012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.743012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:46.743012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:46.743012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:46.743580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.744299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:46.744299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.744299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:46.751440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:46.752095 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:46.752278 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:46.752640 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:29:46.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:46.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.755829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:46.755829 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.755829 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:46.755829 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:46.755829 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.755829 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:46.755829 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.755829 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:46.757267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.757267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:46.757267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:46.757267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:46.757267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.757267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:46.757267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:46.757267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:46.757267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.757267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:46.757267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:46.757267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:46.757267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:46.757267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:46.757267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:46.757804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:46.757804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:46.757804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.757804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:46.757804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:46.757804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.757804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:46.757804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.757804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:46.761401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.761401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:46.761401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:46.761401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:46.761401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.761401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:46.761401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:46.761401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:29:46.761401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:46.761401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:46.761401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.761401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:46.761902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:46.761902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:46.761902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.761902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:46.761902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:46.761902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:46.761902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:46.761902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:46.761902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:46.761902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:46.761902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:46.761902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.761902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:46.761902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:46.762156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.762156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:46.762156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.762156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:46.762861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:46.764342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.764342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:46.764342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:46.764342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:46.764342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:46.764342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:46.764342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.764342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:46.764342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:46.764342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.764342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:46.764342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:46.764342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.764342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:46.764342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:46.764342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:46.764802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:46.765123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.765123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:46.765123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:46.765123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.765123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:46.765123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.765123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:46.765123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:46.765123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:46.765123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:46.767629 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:46.767811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.772166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:46.772166 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.772166 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:46.772166 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:46.772166 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.772166 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:46.772166 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.772166 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:46.773150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.773150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:46.773150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:46.773150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:46.773150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.773150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:46.773150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:46.773150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:46.773150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.773150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:46.773150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:46.773150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:46.773150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:46.773150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:46.773150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:46.774156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:46.774156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:46.774156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.774156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:46.774156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:46.774156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.774156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:46.774156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.774156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:46.778686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.778686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:46.778686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:46.778686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:46.778686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.778686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:46.778686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:46.778686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:46.778686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:46.778686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.778686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:46.779230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:46.779230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:46.779230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.779230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:46.779230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:46.779230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:46.779230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:46.779230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:46.779230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:46.779230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:46.779230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:46.779230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.779230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:46.779230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:46.779591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.779591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:46.779591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.779591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:46.779591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.779591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:46.779591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:46.779591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:46.779591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:46.779591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:46.779591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.779591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:46.779591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:46.779591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:46.779974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:46.781723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:46.781723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:46.781723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:46.781723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:46.781723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.781723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:46.781723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:46.781723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.781723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:46.781723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.781723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:46.781723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:46.781723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:46.782189 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:46.782189 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:46.782189 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:46.784486 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:46.784486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.784912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:46.784912 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.784912 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:46.784912 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:46.784912 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.784912 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:46.784912 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.784912 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:46.785938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.785938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:46.785938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:46.785938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:46.785938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.785938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:46.785938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:46.785938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.785938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:46.785938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:46.785938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:46.785938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.786216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:46.786216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:46.786216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:46.786216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:46.786216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:46.786216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:46.786216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:46.786216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:46.786216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.786216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:46.786216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:46.786216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.786216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:46.786437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.786437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:46.787261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.787261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:46.787261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:46.787261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:46.787261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:46.787261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:46.787261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.787261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:46.787261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:46.787261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.787261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:46.787261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:46.787261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.787261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:46.787261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:46.787261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:46.788075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.788615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:46.788615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.788615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:46.860161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:46.860161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:46.860161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:46.860161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:46.945606 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:46.945606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.946773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:46.946773 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.946773 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:46.946773 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:46.946773 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.946773 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:46.946773 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.946773 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:46.948015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.088 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.948015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:46.948015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:46.948015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:46.948015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.948015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:46.948015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:46.948015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.948015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:46.948015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:46.948015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:46.948015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.948427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:46.948427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:46.948427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:46.948427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:46.948427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:46.948427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:46.948427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:46.948427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:46.948427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.948427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:46.948427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:46.948427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.948427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:46.948891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.948891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:46.949345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.949345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:46.949345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:46.949345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:46.949345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:46.949345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:46.949345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.949345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:46.949345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:46.949345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.949345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:46.949345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:46.949345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.949345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:46.949345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:46.949345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:46.951983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:46.952587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:46.952587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:46.952587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:51.725039 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:51.725521 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:51.725521 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:51.728149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:51.728149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:51.728149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:51.728149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:51.728149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:51.728487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:51.728487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:51.728487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.729087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:51.729087 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.729087 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:51.729087 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:51.729087 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.729087 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:51.729087 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.729087 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:51.731260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.731260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:51.731260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:51.731260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:51.731260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.731260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:51.731260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:51.731260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.731260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:51.731260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:51.731260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:51.731260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.731716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:51.731716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:51.731716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:51.731716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:51.731716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:51.731716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:51.731716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:51.731716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:51.731716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.731716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:51.731716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:51.731716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.731716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:51.732227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.732227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:51.735911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.735911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:51.735911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:51.735911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:51.735911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:51.735911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:51.735911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.735911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:51.735911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:51.735911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.735911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:51.735911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:51.735911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.735911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:51.735911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:51.735911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:51.736215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.736521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:51.736521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.736521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:51.752856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:51.752856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:51.752856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:51.752856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:29:51.753052 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:51.753236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.755035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:51.755755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:51.755755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:51.755755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:51.755755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:51.755755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:51.755755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:51.755755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:51.755755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.755755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:51.755755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:51.755755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.755755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:51.755755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.755755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:51.756560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.756560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:51.756560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:51.756560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:51.756560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.756560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:51.756560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:51.756560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:29:51.756560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:51.756560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:51.756560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.756560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:51.756903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:51.756903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:51.756903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.756903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:51.756903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:51.756903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:51.756903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:51.756903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:51.756903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:51.756903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:51.756903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:51.756903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.756903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:51.756903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:51.757204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.757204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:51.757204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.757204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:51.758013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.758013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:51.758013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:51.758013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:51.758013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:51.758013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:51.758013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.758013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:51.758013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:51.758013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.758013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:51.758013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:51.758013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.758013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:51.758013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:51.758013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:51.758627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:51.759029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.759029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:51.759029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:51.759029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.759029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:51.759029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.759029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:51.764968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:51.765785 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:51.765979 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:51.766144 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:51.766665 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:51.767878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.767878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:51.767878 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.767878 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:51.767878 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:51.767878 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.767878 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:51.767878 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.767878 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:51.768836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.768836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:51.768836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:51.768836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:51.768836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.768836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:51.768836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:51.768836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:51.768836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.768836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:51.768836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:51.768836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:51.768836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:51.768836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:51.768836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:51.769315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:51.769315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:51.769315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.769315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:51.769315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:51.769315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.769315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:51.769315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.769315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:51.770686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.770686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:51.770686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:51.770686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:51.770686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.770686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:51.770686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:51.770686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:51.770686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:51.770686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.770686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:51.771706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:51.771706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:51.771706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.771706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:51.771706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:51.771706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:51.771706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:51.771706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:51.771706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:51.771706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:51.771706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:51.771706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.771706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:51.771706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:51.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:51.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:51.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:51.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:51.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:51.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:51.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:51.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:51.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:51.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:51.773563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:51.774147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:51.774147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:51.774147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:51.774147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:51.774147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.774147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:51.774147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:51.774147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.774147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:51.774147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.774147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:51.780517 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:51.781625 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:51.781625 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:51.781825 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:51.782106 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:51.782422 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:51.782764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.783441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:51.783441 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.783441 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:51.783441 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:51.783441 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.783441 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:51.783441 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.783441 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:51.784834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.784834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:51.784834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:51.784834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:51.784834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.784834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:51.784834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:51.784834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.784834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:51.784834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:51.784834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:51.784834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.785182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:51.785182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:51.785182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:51.785182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:51.785182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:51.785182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:51.785182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:51.785182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:51.785182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.785182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:51.785182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:51.785182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.785182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:51.785524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.785524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:51.785524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.785524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:51.785524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:51.785524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:51.785524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:51.785524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:51.785524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.785524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:51.785524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:51.785524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.785524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:51.785524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:51.785524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:51.786697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.787283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:51.787283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:51.787283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.787283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:51.787283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.787283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:51.858672 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:51.859157 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:51.859262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:51.860255 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:51.941474 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:51.941474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.942357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:51.942357 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.942357 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:51.942357 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:51.942357 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.942357 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:51.942357 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.942357 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:51.943745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.083 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.943745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:51.943745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:51.943745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:51.943745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.943745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:51.943745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:51.943745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.943745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:51.943745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:51.943745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:51.943745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.944133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:51.944133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:51.944133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:51.944133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:51.944133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:51.944133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:51.944133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:51.944133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:51.944133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.944133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:51.944133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:51.944133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.944133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:51.944474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.944474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:51.945069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.945069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:51.945069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:51.945069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:51.945069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:51.945069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:51.945069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.945069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:51.945069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:51.945069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.945069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:51.945069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:51.945069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.945069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:51.945069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:51.945069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:51.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:51.947164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:51.947164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:51.947164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:56.726630 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:56.728664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:56.728664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:56.730394 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:56.731645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:56.732009 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:56.733927 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:56.734112 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:56.734671 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:56.734671 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:56.734950 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:56.735206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.736432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:56.736432 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:56.736432 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:56.736432 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:56.736432 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:56.736432 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:56.736432 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.736432 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:56.737439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.737439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:56.737439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:56.737439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:56.737439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.737439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:56.737439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:56.737439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.737439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:56.737439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:56.737439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:56.737439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.738629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:56.738629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:56.738629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:56.738629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:56.738629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:56.738629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:56.738629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:56.738629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:56.738629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:56.738629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:56.738629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:56.738629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:56.738629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:56.738917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.738917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:56.738917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.738917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:56.738917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:56.738917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:56.738917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:56.738917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:56.738917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.738917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:56.738917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:56.738917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.738917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:56.738917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:56.738917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:56.739243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:56.739700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:56.739700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:56.739700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:56.739700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:56.739700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.739700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:56.752726 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:56.759340 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:56.759637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:56.759803 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:29:56.761911 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:56.762357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.764742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:56.764742 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:56.764742 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:56.764742 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:56.764742 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:56.764742 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:56.764742 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.764742 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:56.765718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.765718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:56.765718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:56.765718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:56.765718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.765718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:56.765718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:56.765718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:56.765718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.765718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:56.765718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:56.765718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:56.765718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:56.765718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:56.765718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:56.766719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:56.766719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:56.766719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:56.766719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:56.766719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:56.766719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:56.766719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:56.766719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.766719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:56.766719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.766719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:56.766719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:56.766719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:56.766719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.767045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:56.767045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:56.767045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:29:56.767045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:56.767045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:56.767045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.767045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:56.767045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:56.767045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:56.767045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.767045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:56.767045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:56.767045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:56.767341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:56.767341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:56.767341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:56.767341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:56.767341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:56.767341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:56.767341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:56.767341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:56.767341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:56.767341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:56.767341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.767341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:56.770412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.770412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:56.770412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:56.770412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:56.770412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:56.770412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:56.770412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.770412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:56.770412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:56.770412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.770412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:56.770412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:56.770412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.770412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:56.770412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:56.770412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:56.772622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:56.773010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:56.773010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:56.773010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:56.773010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:56.773010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:56.773010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.773010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:56.773010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:56.777032 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:56.777032 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:56.777032 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:56.777695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:56.778025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.779331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:56.779331 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:56.779331 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:56.779331 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:56.779331 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:56.779331 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:56.779331 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.779331 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:56.780319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.780319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:29:56.780319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:56.780319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:56.780319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.780319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:56.780319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:56.780319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:29:56.780319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.780319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:29:56.780319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:56.780319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:29:56.780319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:56.780319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:29:56.780319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:29:56.780818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:56.780818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:29:56.780818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:56.780818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:29:56.780818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:29:56.780818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:56.780818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:29:56.780818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.780818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:29:56.781357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.781357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:56.781357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:56.781357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:56.781357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.781357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:56.781357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:56.781357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:56.781357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:56.781357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.781357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:56.782393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:56.782393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:56.782393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.782393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:56.782393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:56.782393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:56.782393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:56.782393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:56.782393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:56.782393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:56.782393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:56.782393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:56.782393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:56.782393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:56.782894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:56.782894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:56.782894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.782894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:56.785178 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.787103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:56.787812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:56.788332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:56.788332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:56.788332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.788332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:56.788332 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:56.788332 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:56.788332 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:56.790360 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:56.791119 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:56.791259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.796710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:56.796710 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:56.796710 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:56.796710 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:56.796710 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:56.796710 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:56.796710 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.796710 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:56.796710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.796710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:56.796710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:56.796710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:56.796710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.796710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:56.796710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:56.797347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.797347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:56.797347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:56.797347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:56.797347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.797347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:56.797347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:56.797347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:56.797347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:56.797347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:56.797347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:56.797347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:56.797347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:56.797347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:56.798385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:56.798385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:56.798385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:56.798385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:56.798385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.798385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:56.798385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.798385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:56.798385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:56.798385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:56.798385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:56.798385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:56.798385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.798385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:56.799275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:56.799896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:56.799896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:56.799896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:56.799896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:56.799896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:56.799896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:56.799896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:56.799896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:56.799896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:56.799896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:29:56.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:29:56.865041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:29:56.865241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:29:56.866103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:29:57.016697 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:29:57.016894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:57.019303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:29:57.019303 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:57.019303 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:29:57.019303 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:29:57.019303 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:57.019303 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:29:57.019303 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:57.019303 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:29:57.022960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.154 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:57.022960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:29:57.022960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:57.022960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:29:57.022960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:57.022960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:57.022960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:57.022960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:57.022960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:57.022960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:57.022960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:29:57.022960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:57.023733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:29:57.023733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:57.023733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:29:57.023733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:57.023733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:29:57.023733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:29:57.023733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:57.023733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:29:57.023733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:57.023733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:29:57.023733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:29:57.023733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:57.023733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:29:57.024116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:57.024116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:29:57.028369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:57.028369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:29:57.028369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:29:57.028369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:29:57.028369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:29:57.028369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:29:57.028369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:57.028369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:29:57.028369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:29:57.028369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:57.028369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:29:57.028369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:29:57.028369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:29:57.028369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:29:57.028369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:29:57.028369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:29:57.029071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:29:57.029639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:29:57.029639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:29:57.029639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:01.731742 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:01.731742 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:01.731742 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:01.734822 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:01.734822 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:01.736380 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:01.736380 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:01.736380 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:01.736380 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:01.736380 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:01.736579 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:01.736622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.738820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:01.738820 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:01.738820 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:01.738820 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:01.738820 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:01.738820 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:01.738820 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.738820 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:01.739639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.739639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:01.739639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:01.739639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:01.739639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.739639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:01.739639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:01.739639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.739639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:01.739639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:01.739639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:01.739639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.739930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:01.739930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:01.739930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:01.739930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:01.739930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:01.739930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:01.739930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:01.739930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:01.739930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:01.739930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:01.739930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:01.739930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:01.739930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:01.740235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.740235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:01.740235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.740235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:01.740235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:01.740235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:01.740235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:01.740235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:01.740235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.740235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:01.740235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:01.740235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.740235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:01.740235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:01.740235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:01.740726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:01.741172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:01.741172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:01.741172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:01.741172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:01.741172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.741172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:01.761001 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:01.764958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:01.764958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:01.764958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:30:01.764958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:01.764958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.773472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:01.773472 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:01.773472 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:01.773472 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:01.773472 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:01.773472 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:01.773472 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.773472 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:01.780220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.780220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:01.780220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:01.780220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:01.780220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.780220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:01.780220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:01.780220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:01.780220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.780220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:01.780220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:01.780220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:01.780220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:01.780220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:01.780220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:01.780802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:01.780802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:01.780802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:01.780802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:01.780802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:01.780802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:01.780802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:01.780802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.780802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:01.780802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.015 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.780802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:01.780802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:01.780802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:01.780802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.781261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:01.781261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:01.781261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:30:01.781261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:01.781261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:01.781261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.781261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:01.781261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:01.781261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:01.781261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.781261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:01.781261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:01.781261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:01.781613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:01.781613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:01.781613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:01.781613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:01.781613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:01.781613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:01.781613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:01.781613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:01.781613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:01.781613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:01.781613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.781613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:01.781958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:01.784126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.784126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:01.784126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:01.784126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:01.784126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:01.784126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:01.784126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.784126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:01.784126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:01.784126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.784126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:01.784126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:01.784126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.784126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:01.784126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:01.784126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:01.784843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:01.785418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:01.785418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:01.785418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:01.785418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:01.785418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:01.785418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.785418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:01.785418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:01.785418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:01.785418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:01.785751 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:01.785751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.786271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:01.786271 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:01.786271 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:01.786271 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:01.786271 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:01.786271 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:01.786271 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.786271 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:01.788969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.788969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:01.788969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:01.788969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:01.788969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.788969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:01.788969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:01.788969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:01.788969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.788969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:01.788969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:01.788969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:01.788969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:01.788969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:01.788969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:01.789705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:01.789705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:01.789705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:01.789705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:01.789705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:01.789705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:01.789705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:01.789705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.789705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:01.789705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:01.789705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.789705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:01.790040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:01.790040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:01.790040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.790040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:01.790040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:01.790040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:01.790040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:01.790040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.790040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:01.790040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:01.790040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:01.790040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.790040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:01.790365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:01.790365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:01.790365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:01.790365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:01.790365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:01.790365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:01.790365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:01.790365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:01.790365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:01.790365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:01.790365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:01.790365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:01.790365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:01.790719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:01.791298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:01.793070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:01.793070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:01.793070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:01.793070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.793070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:01.793070 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:01.793070 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:01.793070 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:01.793070 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:01.793240 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:01.793240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.793240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:01.793240 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:01.793240 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:01.793240 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:01.793240 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:01.793240 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:01.793240 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.793240 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:01.793240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.793240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:01.793240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:01.793240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:01.793475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.793475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:01.793475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:01.793475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.793475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:01.793475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:01.793475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:01.793475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.793475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:01.793475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:01.793475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:01.793475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:01.793475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:01.793475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:01.794231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:01.794231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:01.794231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:01.794231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:01.794231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:01.794231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:01.794231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:01.794231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.794231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:01.794231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.794231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:01.794231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:01.795423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:01.795984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:01.795984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:01.795984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:01.795984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:01.795984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:01.795984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:01.795984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:01.795984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:01.795984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:01.795984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:01.795984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:01.795984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:01.795984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:01.795984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:01.795984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:01.864452 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:01.866475 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:01.866475 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:01.882965 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:02.002759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:02.002930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:02.003568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:02.003568 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:02.003568 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:02.003568 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:02.003568 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:02.003568 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:02.003568 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:02.003568 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:02.006699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.137 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:02.006699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:02.006699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:02.006699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:02.006699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:02.006699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:02.006699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:02.006699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:02.006699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:02.006699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:02.006699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:02.006699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:02.007397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:02.007397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:02.007397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:02.007397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:02.007397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:02.007397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:02.007397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:02.007397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:02.007397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:02.007397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:02.007397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:02.007397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:02.007397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:02.007906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:02.007906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:02.007906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:02.007906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:02.007906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:02.007906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:02.007906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:02.007906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:02.007906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:02.007906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:02.007906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:02.007906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:02.007906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:02.007906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:02.007906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:02.025225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:02.025863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:02.025863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:02.025863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:02.025863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:02.025863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:02.025863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:06.729478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:06.730369 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:06.730369 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:06.731890 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:06.732142 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:06.732352 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:06.732881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:06.733167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:06.733400 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:06.733603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:06.733993 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:06.734173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.735622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:06.735622 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.735622 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:06.735622 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:06.735622 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.735622 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:06.735622 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.735622 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:06.738162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.738162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:06.738162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:06.738162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:06.738162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.738162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:06.738162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:06.738162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.738162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:06.738162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:06.738162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:06.738162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.738886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:06.738886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:06.738886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:06.738886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:06.738886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:06.738886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:06.738886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:06.738886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:06.738886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.738886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:06.738886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:06.738886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.738886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:06.739341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.739341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:06.739341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.739341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:06.739341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:06.739341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:06.739341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:06.739341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:06.739341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.739341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:06.739341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:06.739341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.739341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:06.739341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:06.739341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:06.739717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.740174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:06.740174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:06.740174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.740174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:06.740174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.740174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:06.761926 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:06.762807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:06.762807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:06.762807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:30:06.764557 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:06.764557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.764917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:06.764917 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.764917 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:06.764917 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:06.764917 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.764917 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:06.764917 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.764917 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:06.766039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.766039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:06.766039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:06.766039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:06.766039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.766039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:06.766039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:06.766039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:06.766039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.766039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:06.766039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:06.766039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:06.766039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:06.766039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:06.766039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:06.766509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:06.766509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:06.766509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.766509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:06.766509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:06.766509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.766509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:06.766509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.766509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:06.769997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.769997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:06.769997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:06.769997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:06.769997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.769997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:06.769997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:06.769997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:30:06.769997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:06.769997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:06.769997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.769997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:06.770832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:06.770832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:06.770832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.770832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:06.770832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:06.770832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:06.770832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:06.770832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:06.770832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:06.770832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:06.770832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:06.770832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.770832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:06.770832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:06.771187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.771187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:06.771187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.771187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:06.771867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.771867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:06.771867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:06.771867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:06.771867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:06.771867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:06.771867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.771867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:06.771867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:06.771867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.771867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:06.771867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:06.771867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.771867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:06.771867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:06.771867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:06.772311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:06.772802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.772802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:06.772802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:06.772802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.772802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:06.772802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.772802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:06.784735 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:06.785324 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:06.786223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:06.786494 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:06.787053 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:06.787521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.788228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:06.788228 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.788228 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:06.788228 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:06.788228 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.788228 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:06.788228 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.788228 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:06.790069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.790069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:06.790069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:06.790069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:06.790069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.790069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:06.790069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:06.790069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:06.790069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.790069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:06.790069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:06.790069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:06.790069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:06.790069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:06.790069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:06.790562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:06.790562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:06.790562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.790562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:06.790562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:06.790562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.790562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:06.790562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.790562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:06.791734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:06.792933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.792933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:06.792933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:06.792933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:06.792933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.792933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:06.792933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:06.792933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:06.792933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:06.792933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.792933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:06.793662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:06.793662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:06.793662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.793662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:06.793662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:06.793662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:06.793662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:06.793662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:06.793662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:06.793662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:06.793662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:06.793662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.793662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:06.793662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:06.794280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.794280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:06.794280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.794280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:06.799217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:06.799694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.799694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:06.799694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.799694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:06.801641 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:06.801641 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:06.801641 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:06.801641 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:06.802907 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:06.803431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.807128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:06.807128 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.807128 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:06.807128 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:06.807128 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.807128 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:06.807128 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.807128 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:06.807128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.807128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:06.807128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:06.807128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:06.807128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.807128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:06.807128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:06.808714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.808714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:06.808714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:06.808714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:06.808714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.808714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:06.808714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:06.808714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:06.808714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:06.808714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:06.808714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:06.808714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:06.808714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:06.808714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.809422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:06.809422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:06.809422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.809422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:06.809422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.809422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:06.809422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.809422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:06.809422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:06.809422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:06.809422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:06.809422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:06.809422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.809422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:06.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:06.810740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:06.810740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:06.810740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:06.810740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.810740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:06.810740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:06.810740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.810740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:06.810740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.810740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:06.865796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:06.868280 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:06.868280 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:06.868280 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:06.955037 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:06.955308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.958005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:06.958005 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.958005 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:06.958005 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:06.958005 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.958005 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:06.958005 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.958005 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:06.958005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.089 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.958005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:06.958005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:06.958005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:06.958005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.958005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:06.958005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:06.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:06.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:06.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:06.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:06.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:06.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:06.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:06.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:06.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:06.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:06.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:06.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.959659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:06.959659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:06.959659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.959659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:06.959659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.959659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:06.959659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.959659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:06.959659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:06.959659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:06.959659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:06.959659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:06.959659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.959659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:06.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:06.960632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:06.960632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:06.960632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:06.960632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:06.960632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:06.960632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:06.960632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:06.960632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:06.960632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:11.733804 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:11.735277 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:11.735443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:11.738616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:11.738616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:11.739051 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:11.739167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:11.739225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:11.739383 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:11.739769 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:11.739969 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:11.740120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.740830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:11.740830 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.740830 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:11.740830 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:11.740830 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.740830 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:11.740830 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.740830 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:11.744529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.744529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:11.744529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:11.744529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:11.744529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.744529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:11.744529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:11.744529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.744529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:11.744529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:11.744529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:11.744529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.745639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:11.745639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:11.745639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:11.745639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:11.745639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:11.745639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:11.745639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:11.745639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:11.745639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.745639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:11.745639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:11.745639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.745639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:11.747125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.747125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:11.747125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.747125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:11.747125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:11.747125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:11.747125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:11.747125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:11.747125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.747125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:11.747125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:11.747125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.747125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:11.747125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:11.747125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:11.747900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.749446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:11.749446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:11.749446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.749446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:11.749446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.749446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:11.762668 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:11.765499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:11.765499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:11.765499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:30:11.767613 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:11.767613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.769452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:11.769452 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.769452 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:11.769452 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:11.769452 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.769452 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:11.769452 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.769452 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:11.773462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.773462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:11.773462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:11.773462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:11.773462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.773462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:11.773462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:11.773462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:11.773462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.773462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:11.773462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:11.773462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:11.773462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:11.773462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:11.773462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:11.773968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:11.773968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:11.773968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.773968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:11.773968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:11.773968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.773968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:11.773968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.773968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:11.775357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.775357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:11.775357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:11.775357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:11.775357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.775357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:11.775357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:11.775357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:30:11.775357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:11.775357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:11.775357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.775357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:11.775720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:11.775720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:11.775720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.775720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:11.775720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:11.775720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:11.775720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:11.775720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:11.775720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:11.775720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:11.775720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:11.775720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.775720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:11.775720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:11.777800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.777800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:11.777800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.777800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:11.777800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.777800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:11.777800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:11.777800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:11.777800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:11.777800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:11.777800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.777800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:11.777800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:11.777800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:11.778430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:11.779126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:11.779126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:11.779126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:11.779126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:11.779126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:11.779126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.779126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:11.779126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:11.779126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.779126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:11.779126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.779126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:11.786030 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:11.789043 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:11.791002 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:11.791287 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:11.797631 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:11.798083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.798907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:11.798907 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.798907 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:11.798907 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:11.798907 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.798907 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:11.798907 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.798907 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:11.800740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.800740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:11.800740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:11.800740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:11.800740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.800740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:11.800740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:11.800740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:11.800740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.800740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:11.800740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:11.800740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:11.800740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:11.800740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:11.800740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:11.801188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:11.801188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:11.801188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.801188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:11.801188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:11.801188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.801188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:11.801188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.801188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:11.804228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.804228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:11.804228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:11.804228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:11.804228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.804228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:11.804228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:11.804228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:11.804228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:11.804228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.804228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:11.804613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:11.804613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:11.804613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.804613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:11.804613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:11.804613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:11.804613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:11.804613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:11.804613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:11.804613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:11.804613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:11.804613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.804613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:11.804613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:11.805008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.805008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:11.805008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.805008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.810296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:11.810953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:11.811517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.811517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:11.811517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.811517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:11.812829 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:11.813334 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:11.813443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:11.813755 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:11.822452 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:11.822795 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:11.823434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.826087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:11.826087 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.826087 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:11.826087 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:11.826087 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.826087 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:11.826087 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.826087 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:11.827713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.013 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.827713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:11.827713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:11.827713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:11.827713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.827713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:11.827713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:11.827713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.827713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:11.827713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:11.827713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:11.827713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.828232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:11.828232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:11.828232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:11.828232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:11.828232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:11.828232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:11.828232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:11.828232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:11.828232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.828232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:11.828232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:11.828232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.828232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:11.828858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.828858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:11.828858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.828858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:11.828858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:11.828858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:11.828858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:11.828858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:11.828858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.828858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:11.828858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:11.828858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.828858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:11.828858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:11.828858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:11.829260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.829823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:11.829823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:11.829823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.829823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:11.829823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.829823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:11.867861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:11.868685 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:11.869135 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:11.869705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:11.951278 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:11.951278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.952044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:11.952044 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.952044 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:11.952044 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:11.952044 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.952044 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:11.952044 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.952044 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:11.953111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.083 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.953111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:11.953111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:11.953111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:11.953111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.953111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:11.953111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:11.953111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.953111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:11.953111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:11.953111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:11.953111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.953513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:11.953513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:11.953513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:11.953513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:11.953513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:11.953513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:11.953513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:11.953513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:11.953513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.953513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:11.953513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:11.953513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.953513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:11.953817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.953817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:11.954346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.954346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:11.954346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:11.954346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:11.954346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:11.954346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:11.954346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.954346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:11.954346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:11.954346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.954346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:11.954346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:11.954346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.954346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:11.954346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:11.954346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:11.954801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:11.955226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:11.955226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:11.955226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:16.735844 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:16.736794 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:16.739018 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:16.742114 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:16.742114 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:16.742114 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:16.742114 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:16.742114 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:16.742529 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:16.743729 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:16.743729 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:16.743976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.745522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:16.745522 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.745522 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:16.745522 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:16.745522 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.745522 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:16.745522 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.745522 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:16.745922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.745922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:16.745922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:16.745922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:16.745922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.745922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:16.745922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:16.745922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.745922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:16.745922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:16.745922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:16.745922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.746301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:16.746301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:16.746301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:16.746301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:16.746301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:16.746301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:16.746301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:16.746301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:16.746301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.746301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:16.746301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:16.746301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.746301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:16.746736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.746736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:16.747998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.747998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:16.747998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:16.747998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:16.747998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:16.747998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:16.747998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.747998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:16.747998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:16.747998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.747998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:16.747998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:16.747998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.747998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:16.747998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:16.747998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:16.748711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.749379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:16.749379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.749379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:16.766942 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:16.767943 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:16.768155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:16.768408 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:30:16.770257 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:16.771297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.773390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:16.773390 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.773390 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:16.773390 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:16.773390 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.773390 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:16.773390 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.773390 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:16.773997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.773997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:16.773997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:16.773997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:16.773997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.773997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:16.773997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:16.773997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:16.773997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.773997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:16.773997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:16.773997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:16.773997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:16.773997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:16.773997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:16.775586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:16.775586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:16.775586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.775586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:16.775586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:16.775586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.775586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:16.775586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.775586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:16.775586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.775586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:16.775586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:16.775586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:16.775586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.776554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:16.776554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:16.776554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:30:16.776554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:16.776554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:16.776554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.776554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:16.776554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:16.776554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:16.776554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.776554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:16.776554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:16.776554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:16.776972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:16.776972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:16.776972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:16.776972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:16.776972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:16.776972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.776972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:16.776972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:16.776972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.776972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:16.776972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.776972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:16.777349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.777349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:16.777349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:16.777349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:16.777349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:16.777349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:16.777349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.777349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:16.777349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:16.777349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.777349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:16.777349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:16.777349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.777349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:16.777349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:16.777349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:16.778023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:16.778358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.778358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:16.778358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:16.778358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.778358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:16.778358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.778358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:16.789308 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:16.789928 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:16.790170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:16.790241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:16.794594 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:16.794594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.796119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:16.796119 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.796119 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:16.796119 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:16.796119 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.796119 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:16.796119 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.796119 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:16.797285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.797285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:16.797285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:16.797285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:16.797285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.797285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:16.797285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:16.797285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:16.797285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.797285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:16.797285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:16.797285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:16.797285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:16.797285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:16.797285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:16.798335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:16.798335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:16.798335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.798335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:16.798335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:16.798335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.798335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:16.798335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.798335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:16.798335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.798335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:16.798335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:16.798335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:16.798335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.799337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:16.799337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:16.799337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:16.799337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:16.799337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.799337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:16.799337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:16.799337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:16.799337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.799337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:16.799337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:16.799337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:16.799337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:16.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:16.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:16.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:16.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:16.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:16.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:16.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:16.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.800944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:16.801595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:16.802680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.802680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:16.802680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.802680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:16.813465 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:16.814604 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:16.814833 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:16.816037 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:16.819724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:16.825769 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:16.825769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.825769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:16.825769 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.825769 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:16.825769 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:16.825769 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.825769 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:16.825769 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.825769 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:16.826777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.011 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.826777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:16.826777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:16.826777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:16.826777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.826777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:16.826777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:16.826777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.826777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:16.826777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:16.826777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:16.826777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.827144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:16.827144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:16.827144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:16.827144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:16.827144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:16.827144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:16.827144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:16.827144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:16.827144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.827144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:16.827144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:16.827144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.827144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:16.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:16.832016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.832016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:16.832016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:16.832016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:16.832016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:16.832016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:16.832016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.832016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:16.832016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:16.832016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.832016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:16.832016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:16.832016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.832016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:16.832016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:16.832016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:16.832660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.833193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:16.833193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.833193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:16.869728 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:16.869728 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:16.869902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:16.875002 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:16.936508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:16.937030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.941045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:16.941045 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.941045 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:16.941045 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:16.941045 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.941045 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:16.941045 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.941045 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:16.942001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.071 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.942001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:16.942001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:16.942001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:16.942001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.942001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:16.942001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:16.942001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.942001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:16.942001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:16.942001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:16.942001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.942932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:16.942932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:16.942932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:16.942932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:16.942932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:16.942932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:16.942932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:16.942932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:16.942932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.942932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:16.942932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:16.942932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.942932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:16.943289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.943289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:16.944122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.944122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:16.944122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:16.944122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:16.944122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:16.944122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:16.944122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.944122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:16.944122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:16.944122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.944122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:16.944122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:16.944122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.944122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:16.944122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:16.944122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:16.944621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:16.945208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:16.945208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:16.945208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:21.738552 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:21.738552 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:21.738552 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:21.741026 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:21.741565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:21.742520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:21.743030 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:21.743030 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:21.744272 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:21.744272 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:21.744520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:21.745263 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:21.745927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.747367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:21.747367 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.747367 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:21.747367 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:21.747367 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.747367 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:21.747367 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.747367 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:21.747911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.747911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:21.747911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:21.747911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:21.747911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.747911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:21.747911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:21.747911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.747911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:21.747911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:21.747911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:21.747911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.748482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:21.748482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:21.748482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:21.748482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:21.748482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:21.748482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:21.748482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:21.748482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:21.748482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.748482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:21.748482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:21.748482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.748482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:21.748939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.748939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:21.748939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.748939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:21.748939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:21.748939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:21.748939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:21.748939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:21.748939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.748939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:21.748939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:21.748939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.748939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:21.748939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:21.748939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:21.749470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.750114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:21.750114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:21.750114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.750114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:21.750114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.750114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:21.767896 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:21.768337 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:21.768420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:21.768473 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:30:21.770221 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:21.770221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.771804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:21.771804 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.771804 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:21.771804 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:21.771804 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.771804 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:21.771804 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.771804 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:21.772509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.772509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:21.772509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:21.772509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:21.772509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.772509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:21.772509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:21.772509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:21.772509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.772509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:21.772509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:21.772509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:21.772509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:21.772509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:21.772509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:21.773220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:21.773220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:21.773220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.773220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:21.773220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:21.773220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.773220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:21.773220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.773220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:21.773220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.773220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:21.773220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:21.773220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:21.773220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.773613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:21.773613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:21.773613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:30:21.773613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:21.773613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:21.773613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.773613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:21.773613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:21.773613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:21.773613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.773613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:21.773613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:21.773613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:21.773975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:21.773975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:21.773975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:21.773975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:21.773975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:21.773975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.773975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:21.773975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:21.773975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.773975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:21.773975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.773975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:21.774288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.774288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:21.774288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:21.774288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:21.774288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:21.774288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:21.774288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.774288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:21.774288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:21.774288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.774288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:21.774288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:21.774288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.774288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:21.774288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:21.774288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:21.774700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:21.775258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.775258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:21.775258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:21.775258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.775258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:21.775258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.775258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:21.792218 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:21.792939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:21.793116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:21.793116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:21.795359 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:21.796744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.799704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:21.799704 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.799704 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:21.799704 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:21.799704 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.799704 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:21.799704 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.799704 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:21.803562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.803562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:21.803562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:21.803562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:21.803562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.803562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:21.803562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:21.803562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:21.803562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.803562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:21.803562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:21.803562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:21.803562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:21.803562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:21.803562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:21.804174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:21.804174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:21.804174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.804174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:21.804174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:21.804174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.804174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:21.804174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.804174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:21.804174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.804174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:21.804174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:21.804174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:21.804174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.804423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:21.804423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:21.804423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:21.804423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:21.804423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.804423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:21.804423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:21.804423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:21.804423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.804423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:21.804423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:21.804423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:21.804423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:21.804749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:21.804749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:21.804749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:21.804749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:21.804749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.804749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:21.804749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:21.804749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.804749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:21.804749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.804749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.805378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:21.806228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:21.807150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.807150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:21.807150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.807150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:21.815529 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:21.816143 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:21.816730 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:21.816730 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:21.818414 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:21.818886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:21.819389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.820600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:21.820600 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.820600 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:21.820600 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:21.820600 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.820600 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:21.820600 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.820600 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:21.823175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.823175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:21.823175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:21.823175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:21.823175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.823175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:21.823175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:21.823175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.823175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:21.823175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:21.823175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:21.823175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.823653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:21.823653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:21.823653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:21.823653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:21.823653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:21.823653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:21.823653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:21.823653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:21.823653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.823653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:21.823653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:21.823653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.823653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:21.824001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.824001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:21.824001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.824001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:21.824001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:21.824001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:21.824001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:21.824001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:21.824001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.824001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:21.824001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:21.824001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.824001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:21.824001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:21.824001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:21.824428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.825008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:21.825008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:21.825008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.825008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:21.825008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.825008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:21.872189 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:21.872189 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:21.872189 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:21.872189 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:21.916710 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=246 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:30:21.986603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:21.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.988312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:21.988312 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.988312 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:21.988312 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:21.988312 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.988312 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:21.988312 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.988312 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:21.990442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.118 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.990442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:21.990442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:21.990442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:21.990442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.990442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:21.990442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:21.990442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.990442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:21.990442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:21.990442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:21.990442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.990965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:21.990965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:21.990965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:21.990965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:21.990965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:21.990965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:21.990965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:21.990965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:21.990965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.990965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:21.990965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:21.990965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.990965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:21.991324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.991324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:21.996034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.996034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:21.996034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:21.996034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:21.996034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:21.996034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:21.996034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.996034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:21.996034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:21.996034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.996034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:21.996034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:21.996034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.996034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:21.996034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:21.996034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:21.996682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:21.997656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:21.997656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:21.997656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:26.740324 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:26.740877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:26.740877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:26.741765 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:26.741948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:26.742076 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:26.742196 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:26.742347 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:26.742418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:26.742532 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:26.742676 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:26.743584 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:26.743764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.750241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:26.750241 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.750241 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:26.750241 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:26.750241 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.750241 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:26.750241 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.750241 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:26.750241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.750241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:26.750241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:26.750241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:26.750241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.750241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:26.750241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:26.750907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.750907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:26.750907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:26.750907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:26.750907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.750907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:26.750907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:26.750907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:26.750907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:26.750907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:26.750907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:26.750907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:26.750907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:26.750907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.751761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:26.751761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:26.751761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.751761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:26.751761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.751761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:26.751761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.751761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:26.751761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:26.751761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:26.751761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:26.751761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:26.751761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.751761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:26.752255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:26.752800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:26.752800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:26.752800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:26.752800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.752800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:26.752800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:26.752800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.752800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:26.752800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.752800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:26.770173 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:26.770933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:26.770933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:26.770933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:30:26.771997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:26.772204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.772759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:26.772759 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.772759 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:26.772759 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:26.772759 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.772759 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:26.772759 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.772759 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:26.775938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.775938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:26.775938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:26.775938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:26.775938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.775938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:26.775938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:26.775938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:26.775938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.775938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:26.775938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:26.775938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:26.775938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:26.775938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:26.775938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:26.776466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:26.776466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:26.776466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.776466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:26.776466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:26.776466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.776466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:26.776466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.776466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:26.776466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.776466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:26.776466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:26.776466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:26.776466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.776922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:26.776922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:26.776922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:30:26.776922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:26.776922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:26.776922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.776922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:26.776922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:26.776922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:26.776922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.776922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:26.776922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:26.776922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:26.777281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:26.777281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:26.777281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:26.777281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:26.777281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:26.777281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.777281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:26.777281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:26.777281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.777281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:26.777281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.777281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:26.777719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.777719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:26.777719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:26.777719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:26.777719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:26.777719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:26.777719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.777719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:26.777719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:26.777719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.777719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:26.777719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:26.777719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.777719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:26.777719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:26.777719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:26.778200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:26.778718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.778718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:26.778718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:26.778718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.778718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:26.778718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.778718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:26.792933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:26.793935 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:26.793935 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:26.793935 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:26.794971 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:26.795044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.795779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:26.795779 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.795779 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:26.795779 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:26.795779 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.795779 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:26.795779 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.795779 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:26.798203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.798203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:26.798203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:26.798203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:26.798203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.798203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:26.798203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:26.798203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:26.798203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.798203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:26.798203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:26.798203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:26.798203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:26.798203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:26.798203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:26.798691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:26.798691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:26.798691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.798691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:26.798691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:26.798691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.798691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:26.798691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.798691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:26.800561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.800561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:26.800561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:26.800561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:26.800561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.800561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:26.800561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:26.800561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:26.800561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:26.800561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.800561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:26.801372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:26.801372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:26.801372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.801372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:26.801372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:26.801372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:26.801372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:26.801372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:26.801372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:26.801372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:26.801372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:26.801372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.801372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:26.801372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:26.801781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.801781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:26.801781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.801781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.803530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:26.803970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:26.804412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.804412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:26.804412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.804412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:26.816326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:26.817247 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:26.817247 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:26.817247 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:26.817686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:26.818118 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:26.818452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.819154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:26.819154 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.819154 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:26.819154 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:26.819154 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.819154 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:26.819154 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.819154 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:26.820905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.820905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:26.820905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:26.820905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:26.820905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.820905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:26.820905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:26.820905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.820905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:26.820905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:26.820905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:26.820905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.821418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:26.821418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:26.821418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:26.821418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:26.821418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:26.821418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:26.821418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:26.821418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:26.821418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.821418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:26.821418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:26.821418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.821418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:26.821873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.821873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:26.824457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.824457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:26.824457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:26.824457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:26.824457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:26.824457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:26.824457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.824457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:26.824457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:26.824457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.824457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:26.824457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:26.824457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.824457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:26.824457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:26.824457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:26.824980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.825651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:26.825651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.825651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:26.870213 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:26.871136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:26.871763 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:26.872315 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:26.965718 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:26.965953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.967876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:26.967876 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.967876 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:26.967876 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:26.967876 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.967876 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:26.967876 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.967876 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:26.968906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.097 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.968906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:26.968906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:26.968906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:26.968906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.968906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:26.968906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:26.968906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.968906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:26.968906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:26.968906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:26.968906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.969289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:26.969289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:26.969289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:26.969289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:26.969289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:26.969289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:26.969289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:26.969289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:26.969289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.969289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:26.969289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:26.969289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.969289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:26.969730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.969730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:26.972000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.972000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:26.972000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:26.972000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:26.972000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:26.972000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:26.972000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.972000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:26.972000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:26.972000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.972000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:26.972000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:26.972000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.972000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:26.972000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:26.972000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:26.972523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:26.973251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:26.973251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:26.973251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:31.742576 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:31.743887 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:31.744022 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:31.745229 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:31.745417 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:31.745597 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:31.745763 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:31.745897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:31.746006 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:31.746113 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:31.746210 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:31.746314 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:31.746459 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:31.746660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.747188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:31.747188 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.747188 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:31.747188 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:31.747188 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.747188 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:31.747188 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.747188 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:31.748048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.748048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:31.748048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:31.748048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:31.748048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.748048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:31.748048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:31.748048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.748048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:31.748048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:31.748048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:31.748048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.748676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:31.748676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:31.748676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:31.748676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:31.748676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:31.748676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:31.748676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:31.748676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:31.748676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.748676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:31.748676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:31.748676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.748676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:31.748964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.748964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:31.753213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.753213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:31.753213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:31.753213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:31.753213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:31.753213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:31.753213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.753213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:31.753213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:31.753213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.753213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:31.753213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:31.753213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.753213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:31.753213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:31.753213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:31.753622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.754134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:31.754134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.754134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:31.770749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:31.771220 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:31.771413 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:31.771571 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:30:31.772092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:31.772332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.773053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:31.773053 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.773053 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:31.773053 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:31.773053 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.773053 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:31.773053 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.773053 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:31.776047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.776047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:31.776047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:31.776047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:31.776047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.776047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:31.776047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:31.776047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:31.776047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.776047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:31.776047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:31.776047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:31.776047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:31.776047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:31.776047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:31.777000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:31.777000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:31.777000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.777000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:31.777000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:31.777000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.777000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:31.777000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.777000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:31.777334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.777334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:31.777334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:31.777334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:31.777334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.777334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:31.777334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:31.777334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:30:31.777334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:31.777334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:31.777334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.777334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:31.777659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:31.777659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:31.777659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.777659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:31.777659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:31.777659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:31.777659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:31.777659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:31.777659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:31.777659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:31.777659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:31.777659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.777659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:31.777659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:31.778020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.778020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:31.778020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.778020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:31.778455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.778455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:31.778455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:31.778455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:31.778455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:31.778455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:31.778455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.778455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:31.778455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:31.778455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.778455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:31.778455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:31.778455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.778455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:31.778455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:31.778455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:31.778871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:31.779306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.779306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:31.779306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:31.779306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.779306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:31.779306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.779306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:31.793577 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:31.793967 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:31.794817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:31.794817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:31.794817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:31.794817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.797462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:31.797462 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.797462 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:31.797462 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:31.797462 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.797462 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:31.797462 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.797462 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:31.798618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.798618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:31.798618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:31.798618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:31.798618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.798618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:31.798618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:31.798618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:31.798618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.798618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:31.798618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:31.798618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:31.798618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:31.798618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:31.798618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:31.799157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:31.799157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:31.799157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.799157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:31.799157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:31.799157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.799157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:31.799157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.799157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:31.801563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.801563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:31.801563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:31.801563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:31.801563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.801563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:31.801563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:31.801563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:31.801563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:31.801563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.801563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:31.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:31.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:31.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:31.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:31.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:31.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:31.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:31.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:31.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:31.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:31.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:31.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:31.802440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.802440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:31.802440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.802440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:31.802440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.802440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:31.802440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:31.802440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:31.802440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:31.802440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:31.802440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.802440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:31.802440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:31.802440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:31.802834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:31.803245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:31.803245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:31.803245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:31.803245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:31.803245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.803245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:31.803245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:31.803245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.803245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:31.803245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.803245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:31.818737 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:31.819352 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:31.819581 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:31.819857 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:31.822407 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:31.822691 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:31.822988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.823579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:31.823579 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.823579 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:31.823579 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:31.823579 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.823579 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:31.823579 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.823579 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:31.824633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.824633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:31.824633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:31.824633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:31.824633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.824633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:31.824633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:31.824633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.824633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:31.824633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:31.824633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:31.824633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.824879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:31.824879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:31.824879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:31.824879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:31.824879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:31.824879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:31.824879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:31.824879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:31.824879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.824879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:31.824879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:31.824879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.824879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:31.825091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.825091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:31.825754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.825754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:31.825754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:31.825754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:31.825754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:31.825754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:31.825754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.825754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:31.825754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:31.825754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.825754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:31.825754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:31.825754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.825754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:31.825754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:31.825754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:31.826057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.826382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:31.826382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.826382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:31.872461 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:31.872759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:31.872875 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:31.873730 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:31.947616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:31.947982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.948768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:31.948768 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.948768 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:31.948768 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:31.948768 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.948768 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:31.948768 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.948768 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:31.949809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.076 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.949809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:31.949809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:31.949809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:31.949809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.949809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:31.949809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:31.949809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.949809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:31.949809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:31.949809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:31.949809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.950208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:31.950208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:31.950208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:31.950208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:31.950208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:31.950208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:31.950208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:31.950208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:31.950208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.950208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:31.950208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:31.950208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.950208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:31.950629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.950629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:31.950906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.950906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:31.950906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:31.950906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:31.950906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:31.950906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:31.950906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.950906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:31.950906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:31.950906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.950906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:31.950906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:31.950906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.950906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:31.950906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:31.950906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:31.951482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:31.952093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:31.952093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:31.952093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:36.745908 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:36.749334 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:36.749334 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:36.750040 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:36.750516 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:36.751070 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:36.751352 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:36.751658 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:36.751916 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:36.752158 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:36.752395 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:36.752654 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:36.752880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:36.753118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.753943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:36.753943 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.753943 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:36.753943 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:36.753943 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.753943 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:36.753943 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.753943 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:36.755404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.755404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:36.755404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:36.755404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:36.755404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.755404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:36.755404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:36.755404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.755404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:36.755404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:36.755404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:36.755404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.755907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:36.755907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:36.755907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:36.755907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:36.755907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:36.755907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:36.755907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:36.755907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:36.755907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.755907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:36.755907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:36.755907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.755907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:36.756250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.756250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:36.757918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.757918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:36.757918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:36.757918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:36.757918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:36.757918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:36.757918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.757918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:36.757918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:36.757918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.757918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:36.757918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:36.757918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.757918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:36.757918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:36.757918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:36.758530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.759746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:36.759746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.759746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:36.771917 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:36.773078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:36.773078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:36.773078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:30:36.773621 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:36.774008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.774874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:36.774874 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.774874 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:36.774874 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:36.774874 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.774874 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:36.774874 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.774874 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:36.776016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.776016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:36.776016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:36.776016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:36.776016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.776016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:36.776016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:36.776016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:36.776016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.776016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:36.776016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:36.776016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:36.776016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:36.776016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:36.776016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:36.776387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:36.776387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:36.776387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.776387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:36.776387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:36.776387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.776387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:36.776387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.776387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:36.777502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.777502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:36.777502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:36.777502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:36.777502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.777502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:36.777502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:36.777502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:30:36.777502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:36.777502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:36.777502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.777502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:36.777887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:36.777887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:36.777887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.777887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:36.777887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:36.777887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:36.777887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:36.777887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:36.777887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:36.777887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:36.777887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:36.777887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.777887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:36.777887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:36.778192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.778192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:36.778192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.778192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:36.779662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.779662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:36.779662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:36.779662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:36.779662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:36.779662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:36.779662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.779662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:36.779662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:36.779662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.779662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:36.779662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:36.779662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.779662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:36.779662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:36.779662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:36.780012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:36.780376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.780376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:36.780376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:36.780376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.780376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:36.780376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.780376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:36.794235 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:36.794749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:36.795025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:36.795291 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:36.795831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:36.797230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.798147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:36.798147 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.798147 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:36.798147 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:36.798147 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.798147 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:36.798147 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.798147 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:36.799647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.799647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:36.799647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:36.799647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:36.799647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.799647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:36.799647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:36.799647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:36.799647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.799647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:36.799647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:36.799647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:36.799647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:36.799647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:36.799647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:36.800121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:36.800121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:36.800121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.800121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:36.800121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:36.800121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.800121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:36.800121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.800121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:36.801485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.801485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:36.801485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:36.801485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:36.801485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.801485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:36.801485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:36.801485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:36.801485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:36.801485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.801485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:36.801837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:36.801837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:36.801837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.801837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:36.801837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:36.801837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:36.801837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:36.801837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:36.801837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:36.801837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:36.801837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:36.801837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.801837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:36.801837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:36.802212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.802212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:36.802212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.802212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.803773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:36.804304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:36.804853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.804853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:36.804853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.804853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:36.819437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:36.820746 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:36.820746 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:36.820746 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:36.823679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:36.825713 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:36.826268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.827407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:36.827407 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.827407 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:36.827407 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:36.827407 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.827407 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:36.827407 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.827407 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:36.829160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.829160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:36.829160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:36.829160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:36.829160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.829160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:36.829160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:36.829160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.829160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:36.829160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:36.829160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:36.829160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.830889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:36.830889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:36.830889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:36.830889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:36.830889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:36.830889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:36.830889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:36.830889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:36.830889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.830889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:36.830889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:36.830889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.830889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:36.831452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.831452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:36.833020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.833020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:36.833020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:36.833020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:36.833020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:36.833020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:36.833020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.833020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:36.833020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:36.833020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.833020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:36.833020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:36.833020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.833020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:36.833020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:36.833020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:36.833532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.834087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:36.834087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.834087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:36.872903 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:36.873252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:36.873529 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:36.874472 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:36.936184 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:36.936741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.937018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:36.937018 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.937018 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:36.937018 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:36.937018 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.937018 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:36.937018 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.937018 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:36.937901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.063 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.937901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:36.937901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:36.937901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:36.937901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.937901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:36.937901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:36.937901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.937901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:36.937901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:36.937901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:36.937901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.938391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:36.938391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:36.938391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:36.938391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:36.938391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:36.938391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:36.938391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:36.938391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:36.938391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.938391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:36.938391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:36.938391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.938391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:36.938627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.938627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:36.938932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.938932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:36.938932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:36.938932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:36.938932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:36.938932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:36.938932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.938932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:36.938932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:36.938932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.938932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:36.938932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:36.938932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.938932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:36.938932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:36.938932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:36.939269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:36.939602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:36.939602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:36.939602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:41.748117 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:41.748628 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:41.749558 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:41.749813 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:41.749924 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:41.750039 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:41.750182 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:41.750326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:41.750463 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:41.750620 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:41.750754 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:41.750872 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:41.750954 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:41.751054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.751563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:41.751563 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.751563 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:41.751563 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:41.751563 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.751563 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:41.751563 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.751563 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:41.752377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.752377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:41.752377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:41.752377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:41.752377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.752377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:41.752377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:41.752377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.752377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:41.752377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:41.752377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:41.752377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.752743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:41.752743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:41.752743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:41.752743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:41.752743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:41.752743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:41.752743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:41.752743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:41.752743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.752743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:41.752743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:41.752743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.752743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:41.752983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.752983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:41.753226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.753226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:41.753226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:41.753226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:41.753226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:41.753226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:41.753226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.753226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:41.753226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:41.753226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.753226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:41.753226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:41.753226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.753226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:41.753226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:41.753226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:41.753587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.753984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:41.753984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.753984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:41.772694 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:41.773048 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:41.773241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:41.773398 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:30:41.774010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:41.774204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.774792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:41.774792 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.774792 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:41.774792 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:41.774792 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.774792 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:41.774792 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.774792 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:41.775383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.775383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:41.775383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:41.775383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:41.775383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.775383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:41.775383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:41.775383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:41.775383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.775383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:41.775383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:41.775383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:41.775383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:41.775383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:41.775383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:41.775875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:41.775875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:41.775875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.775875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:41.775875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:41.775875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.775875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:41.775875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.775875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:41.776362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.776362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:41.776362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:41.776362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:41.776362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.776362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:41.776362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:41.776362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:30:41.776362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:41.776362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:41.776362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.776362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:41.776740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:41.776740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:41.776740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.776740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:41.776740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:41.776740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:41.776740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:41.776740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:41.776740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:41.776740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:41.776740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:41.776740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.776740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:41.776740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:41.777190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.777190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:41.777190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.777190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:41.777190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.777190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:41.777190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:41.777190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:41.777190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:41.777190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:41.777190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.777190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:41.777190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:41.777190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:41.777596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:41.778241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:41.778241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:41.778241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:41.778241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:41.778241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:41.778241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.778241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:41.778241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:41.778241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.778241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:41.778241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.778241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:41.794860 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:41.795174 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:41.795281 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:41.795365 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:41.796302 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:41.796520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.797040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:41.797040 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.797040 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:41.797040 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:41.797040 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.797040 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:41.797040 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.797040 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:41.797888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.797888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:41.797888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:41.797888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:41.797888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.797888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:41.797888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:41.797888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:41.797888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.797888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:41.797888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:41.797888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:41.797888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:41.797888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:41.797888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:41.798892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:41.798892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:41.798892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.798892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:41.798892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:41.798892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.798892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:41.798892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.798892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:41.799469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.799469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:41.799469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:41.799469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:41.799469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.799469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:41.799469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:41.799469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:41.799469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:41.799469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.799469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:41.800293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:41.800293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:41.800293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.800293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:41.800293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:41.800293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:41.800293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:41.800293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:41.800293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:41.800293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:41.800293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:41.800293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.800293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:41.800293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:41.800647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.800647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:41.800647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.800647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:41.800647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.800647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:41.800647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:41.800647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:41.800647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:41.800647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:41.800647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.800647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:41.800647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:41.800647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:41.800893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:41.801237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:41.801237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:41.801237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:41.801237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:41.801237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.801237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:41.801237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:41.801237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.801237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:41.801237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.801237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:41.820175 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:41.820406 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:41.820582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:41.820773 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:41.821104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:41.821334 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:41.821593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.822022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:41.822022 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.822022 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:41.822022 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:41.822022 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.822022 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:41.822022 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.822022 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:41.822899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.822899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:41.822899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:41.822899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:41.822899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.822899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:41.822899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:41.822899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.822899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:41.822899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:41.822899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:41.822899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.823118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:41.823118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:41.823118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:41.823118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:41.823118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:41.823118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:41.823118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:41.823118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:41.823118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.823118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:41.823118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:41.823118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.823118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:41.823330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.823330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:41.823879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.823879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:41.823879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:41.823879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:41.823879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:41.823879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:41.823879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.823879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:41.823879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:41.823879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.823879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:41.823879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:41.823879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.823879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:41.823879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:41.823879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:41.824417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.824924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:41.824924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.824924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:41.873454 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:41.874100 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:41.874212 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:41.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:41.931432 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:41.931811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.932588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:41.932588 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.932588 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:41.932588 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:41.932588 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.932588 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:41.932588 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.932588 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:41.933375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.933375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:41.933375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:41.933375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:41.933375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.933375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:41.933375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:41.933375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.933375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:41.933375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:41.933375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:41.933375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.933802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:41.933802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:41.933802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:41.933802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:41.933802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:41.933802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:41.933802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:41.933802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:41.933802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.933802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:41.933802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:41.933802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.933802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:41.934157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.934157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:41.934223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.934223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:41.934223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:41.934223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:41.934223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:41.934223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:41.934223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.934223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:41.934223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:41.934223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.934223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:41.934223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:41.934223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.934223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:41.934223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:41.934223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:41.934605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:41.935044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:41.935044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:41.935044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:46.748623 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:46.750383 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:46.750445 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:46.751418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:46.751640 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:46.752684 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:46.752684 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:46.752684 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:46.752684 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:46.752684 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:46.753583 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:46.753583 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:46.753583 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:46.753583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.753583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:46.753583 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.753583 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:46.753583 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:46.753583 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.753583 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:46.753583 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.753583 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:46.754197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.754197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:46.754197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:46.754197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:46.754197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.754197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:46.754197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:46.754197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.754197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:46.754197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:46.754197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:46.754197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.754808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:46.754808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:46.754808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:46.754808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:46.754808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:46.754808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:46.754808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:46.754808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:46.754808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.754808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:46.754808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:46.754808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.754808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:46.755764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.755764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:46.755764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.755764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:46.755764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:46.755764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:46.755764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:46.755764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:46.755764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.755764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:46.755764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:46.755764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.755764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:46.755764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:46.755764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:46.756395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.756854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:46.756854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:46.756854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.756854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:46.756854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.756854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:46.773177 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:46.774533 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:46.774714 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:46.774958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:30:46.777233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:46.777548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.778183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:46.778183 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.778183 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:46.778183 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:46.778183 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.778183 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:46.778183 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.778183 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:46.778893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.778893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:46.778893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:46.778893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:46.778893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.778893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:46.778893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:46.778893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:46.778893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.778893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:46.778893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:46.778893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:46.778893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:46.778893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:46.778893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:46.779238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:46.779238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:46.779238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.779238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:46.779238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:46.779238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.779238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:46.779238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.779238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:46.779671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.779671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:46.779671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:46.779671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:46.779671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.779671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:46.779671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:46.779671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:30:46.779671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:46.779671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:46.779671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.779671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:46.779984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:46.779984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:46.779984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.779984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:46.779984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:46.779984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:46.779984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:46.779984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:46.779984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:46.779984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:46.779984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:46.779984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.779984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:46.779984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:46.780208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.780208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:46.780208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.780208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:46.783557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.783557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:46.783557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:46.783557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:46.783557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:46.783557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:46.783557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.783557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:46.783557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:46.783557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.783557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:46.783557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:46.783557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.783557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:46.783557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:46.783557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:46.784433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:46.784829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.784829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:46.784829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:46.784829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.784829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:46.784829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.784829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:46.795366 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:46.795757 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:46.795927 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:46.796073 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:46.796699 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:46.798496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.800196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:46.800196 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.800196 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:46.800196 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:46.800196 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.800196 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:46.800196 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.800196 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:46.801915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.801915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:46.801915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:46.801915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:46.801915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.801915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:46.801915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:46.801915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:46.801915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.801915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:46.801915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:46.801915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:46.801915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:46.801915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:46.801915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:46.802365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:46.802365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:46.802365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.802365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:46.802365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:46.802365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.802365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:46.802365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.802365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:46.805854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.805854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:46.805854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:46.805854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:46.805854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.805854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:46.805854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:46.805854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:46.805854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:46.805854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.805854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:46.806382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:46.806382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:46.806382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.806382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:46.806382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:46.806382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:46.806382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:46.806382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:46.806382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:46.806382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:46.806382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:46.806382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.806382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:46.806382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:46.806787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.806787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:46.806787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.806787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.808008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:46.809027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:46.810405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.810405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:46.810405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.810405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:46.820530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:46.822191 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:46.822326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:46.822579 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:46.823067 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:46.823344 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:46.823937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.825078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:46.825078 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.825078 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:46.825078 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:46.825078 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.825078 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:46.825078 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.825078 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:46.825891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.825891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:46.825891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:46.825891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:46.825891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.825891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:46.825891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:46.825891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.825891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:46.825891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:46.825891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:46.825891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.827970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:46.827970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:46.827970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:46.827970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:46.827970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:46.827970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:46.827970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:46.827970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:46.827970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.827970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:46.827970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:46.827970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.827970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:46.828361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.828361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:46.828361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.828361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:46.828361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:46.828361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:46.828361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:46.828361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:46.828361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.828361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:46.828361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:46.828361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.828361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:46.828361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:46.828361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:46.828826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.829309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:46.829309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:46.829309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.829309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:46.829309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.829309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:46.874066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:46.874462 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:46.874600 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:46.875418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:46.934220 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:46.934533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:46.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:46.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:46.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:46.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:46.938404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.061 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.938404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:46.938404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:46.938404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:46.938404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.938404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:46.938404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:46.938404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.938404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:46.938404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:46.938404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:46.938404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.938809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:46.938809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:46.938809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:46.938809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:46.938809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:46.938809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:46.938809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:46.938809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:46.938809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.938809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:46.938809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:46.938809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.938809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:46.939200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.939200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:46.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:46.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:46.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:46.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:46.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:46.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:46.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:46.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:46.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:46.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:46.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:46.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:46.940411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:46.942153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:46.942153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:46.942153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:51.750399 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:51.752727 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:51.752727 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:51.754164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:51.754394 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:51.754603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:51.754751 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:51.756300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:51.756770 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:51.757972 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:51.758151 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:51.758336 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:51.758941 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:51.761264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.761896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:51.761896 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.761896 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:51.761896 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:51.761896 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.761896 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:51.761896 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.761896 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:51.763754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.011 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.763754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:51.763754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:51.763754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:51.763754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.763754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:51.763754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:51.763754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.763754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:51.763754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:51.763754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:51.763754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.764422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:51.764422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:51.764422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:51.764422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:51.764422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:51.764422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:51.764422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:51.764422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:51.764422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.764422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:51.764422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:51.764422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.764422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:51.764883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.764883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:51.766512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.766512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:51.766512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:51.766512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:51.766512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:51.766512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:51.766512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.766512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:51.766512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:51.766512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.766512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:51.766512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:51.766512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.766512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:51.766512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:51.766512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:51.767153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.767827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:51.767827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.767827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:51.774943 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:51.775389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:51.777894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:51.778214 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:30:51.779052 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:51.779346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.780064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:51.780064 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.780064 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:51.780064 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:51.780064 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.780064 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:51.780064 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.780064 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:51.781422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.781422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:51.781422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:51.781422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:51.781422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.781422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:51.781422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:51.781422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:51.781422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.781422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:51.781422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:51.781422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:51.781422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:51.781422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:51.781422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:51.782048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:51.782048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:51.782048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.782048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:51.782048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:51.782048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.782048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:51.782048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.782048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:51.783472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.783472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:51.783472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:51.783472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:51.783472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.783472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:51.783472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:51.783472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:30:51.783472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:51.783472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:51.783472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.783472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:51.783975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:51.783975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:51.783975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.783975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:51.783975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:51.783975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:51.783975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:51.783975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:51.783975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:51.783975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:51.783975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:51.783975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.783975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:51.783975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:51.784443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.784443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:51.784443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.784443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:51.788343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.788343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:51.788343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:51.788343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:51.788343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:51.788343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:51.788343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.788343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:51.788343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:51.788343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.788343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:51.788343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:51.788343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.788343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:51.788343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:51.788343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:51.788936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:51.789483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.789483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:51.789483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:51.789483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.789483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:51.789483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.789483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:51.795856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:51.796902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:51.796902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:51.796902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:51.799061 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:51.799340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.800841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:51.800841 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.800841 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:51.800841 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:51.800841 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.800841 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:51.800841 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.800841 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:51.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:51.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:51.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:51.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:51.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:51.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:51.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:51.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:51.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:51.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:51.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:51.802032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:51.802360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:51.802360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:51.802360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.802360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:51.802360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:51.802360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.802360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:51.802360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.802360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:51.805007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.805007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:51.805007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:51.805007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:51.805007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.805007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:51.805007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:51.805007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:51.805007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:51.805007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.805007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:51.805437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:51.805437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:51.805437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.805437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:51.805437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:51.805437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:51.805437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:51.805437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:51.805437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:51.805437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:51.805437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:51.805437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.805437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:51.805437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:51.805795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.805795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:51.805795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.805795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:51.805795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.805795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:51.805795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:51.805795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:51.805795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:51.805795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:51.805795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.805795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:51.805795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:51.805795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:51.806141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:51.806579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:51.806579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:51.806579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:51.806579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:51.806579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.806579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:51.806579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:51.806579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.806579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:51.806579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.806579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:51.824843 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:51.827848 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:51.828150 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:51.828448 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:51.828949 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:51.829249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:51.829633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.832106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:51.832106 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.832106 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:51.832106 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:51.832106 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.832106 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:51.832106 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.832106 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:51.834092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.834092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:51.834092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:51.834092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:51.834092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.834092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:51.834092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:51.834092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.834092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:51.834092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:51.834092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:51.834092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.834474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:51.834474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:51.834474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:51.834474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:51.834474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:51.834474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:51.834474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:51.834474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:51.834474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.834474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:51.834474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:51.834474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.834474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:51.834809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.834809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:51.836044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.836044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:51.836044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:51.836044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:51.836044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:51.836044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:51.836044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.836044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:51.836044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:51.836044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.836044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:51.836044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:51.836044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.836044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:51.836044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:51.836044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:51.836490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.836991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:51.836991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.836991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:51.874745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:51.879512 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:51.879512 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:51.879512 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:51.953226 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:51.956750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.959245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:51.959245 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.959245 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:51.959245 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:51.959245 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.959245 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:51.959245 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.959245 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:51.959245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.079 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.959245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:51.959245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:51.959245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:51.959245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.959245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:51.959245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:51.959717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.959717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:51.959717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:51.959717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:51.959717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.959717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:51.959717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:51.959717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:51.959717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:51.959717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:51.959717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:51.959717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:51.959717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:51.959717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:51.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:51.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:51.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:51.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:51.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:51.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:51.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:51.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:51.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.960056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:51.960808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:51.961355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:51.961355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:51.961355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:51.961355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:51.961355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:51.961355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:51.961355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:51.961355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:51.961355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:56.751234 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:56.751967 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:56.751967 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:56.756039 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:56.757183 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:56.757573 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:56.757978 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:56.761793 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:56.762280 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:56.763884 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:56.764574 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:56.764574 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:56.767485 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:56.767791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.770480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:56.770480 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.770480 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:56.770480 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:56.770480 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.770480 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:56.770480 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.770480 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:56.771603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.019 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.771603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:56.771603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:56.771603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:56.771603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.771603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:56.771603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:56.771603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.771603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:56.771603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:56.771603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:56.771603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.772003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:56.772003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:56.772003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:56.772003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:56.772003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:56.772003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:56.772003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:56.772003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:56.772003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.772003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:56.772003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:56.772003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.772003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:56.772328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.772328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:56.772754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.772754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:56.772754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:56.772754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:56.772754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:56.772754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:56.772754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.772754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:56.772754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:56.772754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.772754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:56.772754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:56.772754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.772754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:56.772754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:56.772754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:56.773199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.774222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:56.774222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.774222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:56.781602 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:56.781602 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:56.781602 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:56.781602 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:30:56.785982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:56.786230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.787125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:56.787125 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.787125 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:56.787125 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:56.787125 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.787125 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:56.787125 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.787125 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:56.789849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.789849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:56.789849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:56.789849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:56.789849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.789849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:56.789849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:56.789849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:56.789849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.789849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:56.789849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:56.789849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:56.789849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:56.789849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:56.789849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:56.790357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:56.790357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:56.790357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.790357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:56.790357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:56.790357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.790357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:56.790357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.790357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:56.790357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.790357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:56.790357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:56.790357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:56.790357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.790853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:56.790853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:56.790853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:30:56.790853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:56.790853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:56.790853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.790853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:56.790853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:56.790853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:56.790853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.790853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:56.790853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:56.790853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:56.791208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:56.791208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:56.791208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:56.791208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:56.791208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:56.791208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.791208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:56.791208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:56.791208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.791208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:56.791208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.791208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:56.800459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.800459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:56.800459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:56.800459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:56.800459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:56.800459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:56.800459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.800459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:56.800459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:56.800459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.800459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:56.800459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:56.800459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.800459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:56.800459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:56.800459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:56.801067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:56.801497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.801497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:56.801497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:56.801497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.801497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:56.801497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.801497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:56.809484 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:56.810006 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:56.810421 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:56.810668 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:56.816884 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:56.817681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.819258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:56.819258 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.819258 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:56.819258 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:56.819258 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.819258 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:56.819258 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.819258 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:56.820070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.820070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:30:56.820070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:56.820070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:56.820070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.820070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:56.820070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:56.820070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:30:56.820070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.820070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:30:56.820070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:56.820070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:30:56.820070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:56.820070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:30:56.820070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:30:56.820493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:56.820493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:30:56.820493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.820493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:30:56.820493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:30:56.820493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.820493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:30:56.820493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.820493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:30:56.823710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.011 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.823710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:56.823710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:56.823710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:56.823710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.823710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:56.823710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:56.823710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:56.823710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:56.823710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.823710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:56.824193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:56.824193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:56.824193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.824193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:56.824193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:56.824193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:56.824193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:56.824193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:56.824193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:56.824193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:56.824193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:56.824193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.824193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:56.824193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:56.824568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.824568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:56.824568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.824568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.827703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:56.828511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:56.828953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.828953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:56.828953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.828953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:56.835413 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:56.836262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:56.836891 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:56.840271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:56.842508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:56.842912 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:56.845138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.849347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:56.849347 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.849347 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:56.849347 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:56.849347 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.849347 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:56.849347 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.849347 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:56.849347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.849347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:56.849347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:56.849347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:56.849347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.849347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:56.849347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:56.850036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.850036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:56.850036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:56.850036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:56.850036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.850036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:56.850036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:56.850036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:56.850036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:56.850036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:56.850036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:56.850036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:56.850036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:56.850036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.850353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:56.850353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:56.850353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.850353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:56.850353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.850353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:56.850353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.850353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:56.850353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:56.850353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:56.850353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:56.850353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:56.850353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.850353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:56.850704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:56.851141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:56.851141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:56.851141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:56.851141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.851141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:56.851141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:56.851141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.851141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:56.851141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.851141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:30:56.879041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:30:56.879421 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:30:56.879776 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:30:56.880993 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:30:56.957250 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:30:56.957522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.958494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:30:56.958494 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.958494 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:30:56.958494 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:30:56.958494 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.958494 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:30:56.958494 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.958494 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:30:56.962355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.078 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.962355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:30:56.962355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:56.962355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:30:56.962355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.962355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:56.962355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:56.962355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.962355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:56.962355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:56.962355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:30:56.962355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.962828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:30:56.962828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:56.962828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:30:56.962828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:56.962828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:30:56.962828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:30:56.962828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:56.962828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:30:56.962828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.962828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:30:56.962828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:30:56.962828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.962828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:30:56.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:30:56.963916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.963916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:30:56.963916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:30:56.963916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:30:56.963916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:30:56.963916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:30:56.963916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.963916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:30:56.963916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:30:56.963916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.963916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:30:56.963916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:30:56.963916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.963916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:30:56.963916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:30:56.963916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:30:56.964365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:30:56.964862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:30:56.964862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:30:56.964862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:01.751820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:01.752704 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:01.756506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:01.760425 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:01.760701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:01.760882 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:01.761038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:01.761172 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:01.761299 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:01.761420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:01.761571 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:01.761690 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:01.761801 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:01.761928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.763964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:01.763964 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.763964 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:01.763964 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:01.763964 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.763964 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:01.763964 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.763964 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:01.767604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.767604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:01.767604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:01.767604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:01.767604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.767604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:01.767604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:01.767604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.767604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:01.767604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:01.767604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:01.767604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.768161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:01.768161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:01.768161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:01.768161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:01.768161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:01.768161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:01.768161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:01.768161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:01.768161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.768161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:01.768161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:01.768161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.768161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:01.768478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.768478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:01.773899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.773899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:01.773899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:01.773899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:01.773899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:01.773899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:01.773899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.773899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:01.773899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:01.773899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.773899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:01.773899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:01.773899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.773899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:01.773899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:01.773899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:01.774640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.775092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:01.775092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.775092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:01.780777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:01.781796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:01.784773 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:01.784985 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:31:01.785667 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:01.785835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.786505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:01.786505 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.786505 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:01.786505 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:01.786505 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.786505 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:01.786505 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.786505 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:01.787732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.787732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:01.787732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:01.787732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:01.787732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.787732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:01.787732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:01.787732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:01.787732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.787732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:01.787732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:01.787732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:01.787732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:01.787732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:01.787732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:01.788273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:01.788273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:01.788273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.788273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:01.788273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:01.788273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.788273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:01.788273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.788273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:01.788464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.788464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:01.788464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:01.788464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:01.788464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.788464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:01.788464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:01.788464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:31:01.788464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:01.788464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:01.788464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.788464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:01.788778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:01.788778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:01.788778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.788778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:01.788778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:01.788778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:01.788778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:01.788778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:01.788778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:01.788778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:01.788778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:01.788778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.788778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:01.788778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:01.789008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.789008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:01.789008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.789008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:01.791004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.791004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:01.791004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:01.791004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:01.791004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:01.791004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:01.791004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.791004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:01.791004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:01.791004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.791004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:01.791004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:01.791004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.791004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:01.791004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:01.791004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:01.791696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:01.792336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.792336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:01.792336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:01.792336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.792336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:01.792336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.792336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:01.811518 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:01.813513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:01.813513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:01.813513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:01.813851 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:01.814310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.814789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:01.814789 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.814789 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:01.814789 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:01.814789 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.814789 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:01.814789 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.814789 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:01.816991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.816991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:01.816991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:01.816991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:01.816991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.816991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:01.816991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:01.816991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:01.816991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.816991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:01.816991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:01.816991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:01.816991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:01.816991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:01.816991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:01.817287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:01.817287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:01.817287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.817287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:01.817287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:01.817287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.817287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:01.817287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.817287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:01.819244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.819244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:01.819244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:01.819244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:01.819244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.819244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:01.819244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:01.819244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:01.819244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:01.819244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.819244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:01.819587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:01.819587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:01.819587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.819587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:01.819587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:01.819587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:01.819587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:01.819587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:01.819587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:01.819587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:01.819587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:01.819587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.819587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:01.819587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:01.820188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.820188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:01.820188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.820188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.822025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:01.822961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:01.823406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.823406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:01.823406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.823406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:01.836292 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:01.836685 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:01.837099 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:01.837099 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:01.837995 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:01.838189 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:01.838423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.839200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:01.839200 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.839200 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:01.839200 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:01.839200 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.839200 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:01.839200 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.839200 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:01.840315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.840315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:01.840315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:01.840315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:01.840315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.840315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:01.840315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:01.840315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.840315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:01.840315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:01.840315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:01.840315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.840911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:01.840911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:01.840911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:01.840911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:01.840911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:01.840911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:01.840911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:01.840911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:01.840911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.840911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:01.840911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:01.840911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.840911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:01.841183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.841183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:01.842282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.842282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:01.842282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:01.842282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:01.842282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:01.842282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:01.842282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.842282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:01.842282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:01.842282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.842282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:01.842282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:01.842282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.842282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:01.842282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:01.842282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:01.842812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.843185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:01.843185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.843185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:01.879576 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:01.880044 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:01.880323 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:01.881728 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:01.955849 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:01.956051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.956650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:01.956650 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.956650 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:01.956650 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:01.956650 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.956650 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:01.956650 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.956650 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:01.959085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.076 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.959085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:01.959085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:01.959085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:01.959085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.959085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:01.959085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:01.959085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.959085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:01.959085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:01.959085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:01.959085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.959391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:01.959391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:01.959391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:01.959391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:01.959391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:01.959391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:01.959391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:01.959391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:01.959391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.959391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:01.959391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:01.959391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.959391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:01.959612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.959612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:01.959612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.959612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:01.959612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:01.959612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:01.959612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:01.959612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:01.959612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.959612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:01.959612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:01.959612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.959612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:01.959612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:01.959612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:01.959981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:01.960492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:01.960492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:01.960492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:01.960492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:01.960492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:01.960492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:06.752272 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:06.753115 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:06.753777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:06.754611 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:06.754934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:06.755155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:06.755337 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:06.755500 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:06.756390 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:06.756390 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:06.756390 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:06.756390 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:06.756390 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:06.756823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.757165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:06.757165 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.757165 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:06.757165 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:06.757165 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.757165 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:06.757165 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.757165 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:06.758962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.758962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:06.758962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:06.758962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:06.758962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.758962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:06.758962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:06.758962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.758962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:06.758962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:06.758962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:06.758962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.760632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:06.760632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:06.760632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:06.760632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:06.760632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:06.760632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:06.760632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:06.760632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:06.760632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.760632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:06.760632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:06.760632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.760632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:06.761052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.761052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:06.761052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.761052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:06.761052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:06.761052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:06.761052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:06.761052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:06.761052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.761052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:06.761052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:06.761052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.761052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:06.761052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:06.761052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:06.761509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.762222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:06.762222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:06.762222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.762222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:06.762222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.762222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:06.784565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:06.784565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:06.784565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:06.784565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:31:06.784565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:06.784955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.785870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:06.785870 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.785870 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:06.785870 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:06.785870 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.785870 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:06.785870 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.785870 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:06.786555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.786555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:06.786555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:06.786555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:06.786555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.786555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:06.786555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:06.786555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:06.786555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.786555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:06.786555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:06.786555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:06.786555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:06.786555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:06.786555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:06.787688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:06.787688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:06.787688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.787688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:06.787688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:06.787688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.787688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:06.787688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.787688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:06.788335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.788335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:06.788335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:06.788335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:06.788335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.788335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:06.788335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:06.788335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:31:06.788335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:06.788335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:06.788335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.788335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:06.789190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:06.789190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:06.789190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.789190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:06.789190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:06.789190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:06.789190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:06.789190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:06.789190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:06.789190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:06.789190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:06.789190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.789190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:06.789190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:06.791276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.791276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:06.791276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.791276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:06.791276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.791276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:06.791276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:06.791276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:06.791276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:06.791276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:06.791276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.791276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:06.791276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:06.791276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:06.792131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:06.792716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:06.792716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:06.792716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:06.792716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:06.792716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:06.792716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.792716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:06.792716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:06.792716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.792716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:06.792716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.792716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:06.814124 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:06.814124 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:06.814124 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:06.814124 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:06.814497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:06.815124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.815895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:06.815895 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.815895 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:06.815895 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:06.815895 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.815895 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:06.815895 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.815895 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:06.817908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.817908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:06.817908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:06.817908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:06.817908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.817908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:06.817908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:06.817908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:06.817908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.817908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:06.817908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:06.817908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:06.817908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:06.817908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:06.817908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:06.818550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:06.818550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:06.818550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.818550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:06.818550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:06.818550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.818550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:06.818550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.818550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:06.820419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.820419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:06.820419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:06.820419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:06.820419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.820419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:06.820419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:06.820419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:06.820419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:06.820419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.820419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:06.820990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:06.820990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:06.820990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.820990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:06.820990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:06.820990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:06.820990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:06.820990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:06.820990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:06.820990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:06.820990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:06.820990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.820990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:06.820990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:06.821408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.821408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:06.821408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.821408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.821571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:06.822133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:06.822767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.822767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:06.822767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.822767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:06.837617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:06.838478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:06.838478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:06.838478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:06.839980 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:06.840363 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:06.840899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.844886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:06.844886 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.844886 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:06.844886 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:06.844886 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.844886 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:06.844886 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.844886 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:06.845774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.845774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:06.845774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:06.845774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:06.845774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.845774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:06.845774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:06.845774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.845774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:06.845774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:06.845774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:06.845774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.846314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:06.846314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:06.846314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:06.846314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:06.846314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:06.846314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:06.846314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:06.846314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:06.846314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.846314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:06.846314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:06.846314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.846314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:06.846756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.846756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:06.848262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.848262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:06.848262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:06.848262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:06.848262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:06.848262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:06.848262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.848262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:06.848262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:06.848262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.848262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:06.848262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:06.848262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.848262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:06.848262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:06.848262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:06.848836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.849457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:06.849457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.849457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:06.880965 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:06.883397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:06.883397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:06.883397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:06.990077 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:06.990821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.992973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:06.992973 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.992973 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:06.992973 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:06.992973 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.992973 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:06.992973 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.992973 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:06.994246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.111 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.994246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:06.994246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:06.994246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:06.994246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.994246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:06.994246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:06.994246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.994246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:06.994246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:06.994246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:06.994246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.994888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:06.994888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:06.994888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:06.994888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:06.994888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:06.994888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:06.994888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:06.994888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:06.994888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.994888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:06.994888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:06.994888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.994888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:06.995278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.995278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:06.996244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.996244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:06.996244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:06.996244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:06.996244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:06.996244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:06.996244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.996244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:06.996244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:06.996244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.996244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:06.996244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:06.996244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.996244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:06.996244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:06.996244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:06.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:06.997291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:06.997291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:06.997291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:11.753018 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:11.755998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:11.762176 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:11.769445 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:11.769445 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:11.774941 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:11.776977 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:11.779391 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:11.781530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:11.781530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:11.784958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:11.785723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:11.788879 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:11.788879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.793674 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:11.793674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:11.793674 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.793674 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:11.793674 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:11.793674 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.793674 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:11.793674 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.793674 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:11.796655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.796655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:11.796655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:11.796655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:11.796655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.796655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:11.796655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:11.796655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.796655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:11.796655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:11.796655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:11.796655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.797138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:11.797138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:11.797138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:11.797138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:11.797138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:11.797138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:11.797138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:11.797138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:11.797138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.797138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:11.797138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:11.797138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.797138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:11.797518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.797518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:11.798731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.798731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:11.798731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:11.798731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:11.798731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:11.798731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:11.798731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.798731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:11.798731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:11.798731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.798731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:11.798731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:11.798731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.798731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:11.798731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:11.798731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:11.799244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.799888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:11.799888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.799888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:11.800051 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:11.800277 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:11.800490 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:31:11.801035 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:11.801275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.801950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:11.801950 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.801950 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:11.801950 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:11.801950 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.801950 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:11.801950 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.801950 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:11.803104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.803104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:11.803104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:11.803104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:11.803104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.803104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:11.803104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:11.803104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:11.803104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.803104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:11.803104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:11.803104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:11.803104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:11.803104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:11.803104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:11.803527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:11.803527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:11.803527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.803527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:11.803527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:11.803527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.803527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:11.803527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.803527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:11.804827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.804827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:11.804827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:11.804827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:11.804827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.804827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:11.804827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:11.804827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:31:11.804827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:11.804827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:11.804827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.804827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:11.805186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:11.805186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:11.805186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.805186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:11.805186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:11.805186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:11.805186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:11.805186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:11.805186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:11.805186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:11.805186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:11.805186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.805186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:11.805186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:11.805523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.805523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:11.805523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.805523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:11.806513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.806513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:11.806513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:11.806513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:11.806513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:11.806513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:11.806513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.806513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:11.806513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:11.806513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.806513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:11.806513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:11.806513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.806513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:11.806513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:11.806513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:11.806981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:11.807425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.807425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:11.807425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:11.807425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.807425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:11.807425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.807425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:11.814366 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:11.814366 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:11.815311 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:11.815489 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:11.816126 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:11.816492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.817208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:11.817208 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.817208 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:11.817208 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:11.817208 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.817208 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:11.817208 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.817208 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:11.818176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.818176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:11.818176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:11.818176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:11.818176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.818176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:11.818176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:11.818176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:11.818176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.818176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:11.818176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:11.818176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:11.818176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:11.818176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:11.818176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:11.818587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:11.818587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:11.818587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.818587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:11.818587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:11.818587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.818587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:11.818587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.818587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:11.822068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.822068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:11.822068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:11.822068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:11.822068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.822068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:11.822068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:11.822068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:11.822068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:11.822068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.822068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:11.822470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:11.822470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:11.822470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.822470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:11.822470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:11.822470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:11.822470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:11.822470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:11.822470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:11.822470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:11.822470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:11.822470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.822470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:11.822470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:11.822847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.822847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:11.822847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.822847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.824140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:11.824639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:11.825193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.825193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:11.825193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.825193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:11.838213 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:11.838644 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:11.838828 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:11.839020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:11.839622 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:11.840263 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:11.840606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.841453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:11.841453 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.841453 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:11.841453 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:11.841453 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.841453 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:11.841453 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.841453 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:11.842840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.842840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:11.842840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:11.842840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:11.842840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.842840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:11.842840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:11.842840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.842840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:11.842840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:11.842840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:11.842840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.843182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:11.843182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:11.843182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:11.843182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:11.843182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:11.843182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:11.843182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:11.843182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:11.843182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.843182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:11.843182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:11.843182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.843182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:11.843503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.843503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:11.844655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.844655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:11.844655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:11.844655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:11.844655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:11.844655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:11.844655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.844655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:11.844655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:11.844655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.844655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:11.844655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:11.844655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.844655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:11.844655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:11.844655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:11.845083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.845500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:11.845500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.845500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:11.882001 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:11.882352 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:11.882598 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:11.883624 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:11.948440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:11.948680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.949289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:11.949289 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.949289 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:11.949289 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:11.949289 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.949289 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:11.949289 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.949289 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:11.950457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.067 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.950457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:11.950457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:11.950457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:11.950457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.950457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:11.950457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:11.950457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.950457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:11.950457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:11.950457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:11.950457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.950987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:11.950987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:11.950987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:11.950987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:11.950987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:11.950987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:11.950987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:11.950987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:11.950987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.950987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:11.950987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:11.950987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.950987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:11.951332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.951332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:11.951800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.951800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:11.951800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:11.951800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:11.951800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:11.951800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:11.951800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.951800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:11.951800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:11.951800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.951800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:11.951800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:11.951800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.951800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:11.951800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:11.951800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:11.952427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:11.953030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:11.953030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:11.953030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:16.754666 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:16.765578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:16.765578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:16.768666 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:16.768666 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:16.768901 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:16.768901 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:16.769071 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:16.769071 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:16.776194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:16.776194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:16.776372 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:16.777438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:16.777438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.777807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:16.777807 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:16.777807 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:16.777807 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:16.777807 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:16.777807 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:16.777807 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.777807 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:16.782342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.015 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.782342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:16.782342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:16.782342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:16.782342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.782342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:16.782342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:16.782342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.782342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:16.782342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:16.782342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:16.782342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.783708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:16.783708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:16.783708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:16.783708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:16.783708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:16.783708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:16.783708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:16.783708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:16.783708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:16.783708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:16.783708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:16.783708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:16.783708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:16.784063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.784063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:16.787964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.787964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:16.787964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:16.787964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:16.787964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:16.787964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:16.787964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.787964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:16.787964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:16.787964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.787964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:16.787964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:16.787964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.787964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:16.787964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:16.787964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:16.788794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:16.789556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:16.789556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.789556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:16.803889 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:16.804020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:16.804146 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:16.804820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:31:16.804820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:16.805129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.811904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:16.811904 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:16.811904 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:16.811904 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:16.811904 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:16.811904 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:16.811904 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.811904 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:16.812884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.812884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:16.812884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:16.812884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:16.812884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.812884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:16.812884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:16.812884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:16.812884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.812884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:16.812884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:16.812884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:16.812884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:16.812884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:16.812884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:16.813330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:16.813330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:16.813330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:16.813330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:16.813330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:16.813330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:16.813330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:16.813330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.813330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:16.837568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.837568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:16.837568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:16.837568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:16.837568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.837568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:16.837568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:16.837568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:31:16.837568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:16.837568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:16.837568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.837568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:16.839432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:16.839432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:16.839432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.839432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:16.839432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:16.839432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:16.839432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:16.839432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:16.839432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:16.839432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:16.839432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:16.839432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:16.839432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:16.839432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:16.841496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:16.841496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:16.841496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.841496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:16.841496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.841496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:16.841496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:16.841496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:16.841496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:16.841496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:16.841496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.841496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:16.841496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:16.841496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:16.841990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:16.842481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:16.842481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:16.842481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:16.842481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:16.842481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:16.842481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:16.842481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:16.842481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:16.842481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:16.842481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:16.842481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.842481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:16.842481 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:16.842481 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:16.844362 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:16.844444 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:16.844526 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:16.845264 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:16.849317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.850257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:16.850257 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:16.850257 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:16.850257 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:16.850257 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:16.850257 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:16.850257 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.850257 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:16.851636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.851636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:16.851636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:16.851636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:16.851636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.851636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:16.851636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:16.851636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:16.851636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.851636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:16.851636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:16.851636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:16.851636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:16.851636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:16.851636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:16.852609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:16.852609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:16.852609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:16.852609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:16.852609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:16.852609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:16.852609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:16.852609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.852609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:16.853803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.853803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:16.853803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:16.853803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:16.853803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.853803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:16.853803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:16.853803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:16.853803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:16.853803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.853803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:16.855750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:16.855750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:16.855750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.855750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:16.855750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:16.855750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:16.855750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:16.855750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:16.855750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:16.855750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:16.855750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:16.855750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:16.855750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:16.855750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:16.857284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:16.857284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:16.857284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.857284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:16.857284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.857284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:16.857284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:16.857284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:16.857284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:16.857284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:16.857284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.857284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:16.857284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:16.857284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:16.857948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:16.858642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:16.858642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:16.858642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:16.858642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:16.858642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:16.858642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:16.858642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:16.858642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:16.858642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:16.858642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.858642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:16.858642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:16.858642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:16.859011 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:16.859011 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:16.859011 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:16.860929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.863500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:16.863500 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:16.863500 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:16.863500 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:16.863500 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:16.863500 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:16.863500 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.863500 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:16.868341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.868341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:16.868341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:16.868341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:16.868341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.868341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:16.868341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:16.868341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.868341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:16.868341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:16.868341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:16.868341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.868805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:16.868805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:16.868805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:16.868805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:16.868805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:16.868805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:16.868805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:16.868805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:16.868805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:16.868805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:16.868805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:16.868805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:16.868805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:16.869146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.869146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:16.874244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.874244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:16.874244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:16.874244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:16.874244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:16.874244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:16.874244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.874244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:16.874244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:16.874244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.874244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:16.874244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:16.874244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.874244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:16.874244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:16.874244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:16.874785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:16.875376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:16.875376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:16.875376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:16.883214 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:16.886655 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:16.886848 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:16.887830 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:17.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:17.043857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:17.046740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:17.046740 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:17.046740 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:17.046740 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:17.046740 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:17.046740 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:17.046740 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:17.046740 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:17.047182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.159 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:17.047182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:17.047182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:17.047182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:17.047182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:17.047182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:17.047182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:17.047182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:17.047182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:17.047182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:17.047182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:17.047182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:17.047581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:17.047581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:17.047581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:17.047581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:17.047581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:17.047581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:17.047581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:17.047581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:17.047581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:17.047581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:17.047581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:17.047581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:17.047581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:17.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:17.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:17.053579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:17.053579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:17.053579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:17.053579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:17.053579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:17.053579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:17.053579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:17.053579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:17.053579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:17.053579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:17.053579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:17.053579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:17.053579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:17.053579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:17.053579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:17.053579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:17.054147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:17.054616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:17.054616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:17.054616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:21.765641 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:21.767665 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:21.767665 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:21.769530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:21.769907 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:21.770046 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:21.771027 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:21.771027 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:21.771027 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:21.771027 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:21.771027 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:21.771406 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:21.771406 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:21.771406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.772368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:21.772368 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.772368 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:21.772368 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:21.772368 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.772368 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:21.772368 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.772368 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:21.777519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.777519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:21.777519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:21.777519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:21.777519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.777519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:21.777519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:21.777519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.777519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:21.777519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:21.777519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:21.777519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.777994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:21.777994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:21.777994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:21.777994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:21.777994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:21.777994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:21.777994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:21.777994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:21.777994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.777994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:21.777994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:21.777994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.777994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:21.778311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.778311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:21.782389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.782389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:21.782389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:21.782389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:21.782389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:21.782389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:21.782389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.782389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:21.782389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:21.782389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.782389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:21.782389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:21.782389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.782389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:21.782389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:21.782389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:21.783044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.785060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:21.785060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.785060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:21.804127 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:21.805837 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:21.805837 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:21.805837 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:31:21.806087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:21.806272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.808239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:21.809103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:21.809103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:21.809103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:21.809103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:21.809103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:21.809103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:21.809103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:21.809103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.809103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:21.809103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:21.809103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.809103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:21.809103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.809103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:21.810306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.810306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:21.810306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:21.810306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:21.810306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.810306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:21.810306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:21.810306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:31:21.810306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:21.810306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:21.810306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.810306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:21.811031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:21.811031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:21.811031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.811031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:21.811031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:21.811031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:21.811031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:21.811031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:21.811031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:21.811031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:21.811031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:21.811031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.811031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:21.811031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:21.812153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.812153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:21.812153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.812153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:21.813639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.813639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:21.813639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:21.813639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:21.813639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:21.813639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:21.813639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.813639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:21.813639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:21.813639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.813639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:21.813639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:21.813639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.813639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:21.813639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:21.813639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:21.814335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:21.815297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.815297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:21.815297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:21.815297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.815297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:21.815297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.815297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:21.842497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:21.844142 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:21.844956 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:21.846039 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:21.846428 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:21.847228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.848167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:21.848167 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.848167 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:21.848167 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:21.848167 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.848167 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:21.848167 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.848167 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:21.856486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.856486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:21.856486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:21.856486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:21.856486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.856486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:21.856486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:21.856486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:21.856486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.856486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:21.856486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:21.856486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:21.856486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:21.856486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:21.856486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:21.858078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:21.858078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:21.858078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.858078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:21.858078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:21.858078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.858078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:21.858078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.858078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:21.858078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.858078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:21.858078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:21.858078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:21.858078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.858555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:21.858555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:21.858555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:21.858555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:21.858555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.858555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:21.858555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:21.858555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:21.858555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.858555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:21.858555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:21.858555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:21.858555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:21.859143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:21.859143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:21.859143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:21.859143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:21.859143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.859143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:21.859143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:21.859143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.859143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:21.859143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.859143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:21.859143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:21.860695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.861193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:21.861831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:21.861831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:21.861831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:21.861831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:21.861831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:21.863281 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:21.864594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.865413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:21.865413 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.865413 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:21.865413 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:21.865413 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.865413 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:21.865413 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.865413 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:21.867066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.867066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:21.867066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:21.867066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:21.867066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.867066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:21.867066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:21.867066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.867066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:21.867066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:21.867066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:21.867066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.869478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:21.869478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:21.869478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:21.869478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:21.869478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:21.869478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:21.869478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:21.869478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:21.869478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.869478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:21.869478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:21.869478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.869478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:21.869842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.869842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:21.869842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.869842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:21.869842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:21.869842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:21.869842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:21.869842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:21.869842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.869842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:21.869842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:21.869842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.869842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:21.869842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:21.869842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:21.870212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.870761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:21.870761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:21.870761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.870761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:21.870761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.870761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:21.886950 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:21.887499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:21.889419 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:21.889419 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:21.925759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=243 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:31:21.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:21.970601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.970601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:21.970601 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.970601 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:21.970601 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:21.970601 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.970601 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:21.970601 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.970601 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:21.970601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.081 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.970601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:21.970601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:21.970601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:21.970601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.970601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:21.971187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:21.971187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.971187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:21.971187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:21.971187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:21.971187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.971187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:21.971187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:21.971187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:21.971187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:21.971187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:21.971187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:21.971187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:21.971187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:21.971726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.971726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:21.971726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:21.971726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.971726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:21.971726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.971726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:21.971726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.971726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:21.971726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:21.971726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:21.971726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:21.971726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:21.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:21.972663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:21.972663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:21.972663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:21.972663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:21.972663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:21.972663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:21.972663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:21.972663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:21.972663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:21.972663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:21.972663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:21.972663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:26.766588 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:26.767810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:26.767936 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:26.769367 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:26.769570 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:26.769807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:26.769871 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:26.770149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:26.770149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:26.770257 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:26.770402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:26.770561 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:26.770686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:26.770842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.771370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:26.771370 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.771370 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:26.771370 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:26.771370 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.771370 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:26.771370 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.771370 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:26.772084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.772084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:26.772084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:26.772084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:26.772084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.772084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:26.772084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:26.772084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.772084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:26.772084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:26.772084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:26.772084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.772411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:26.772411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:26.772411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:26.772411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:26.772411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:26.772411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:26.772411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:26.772411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:26.772411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.772411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:26.772411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:26.772411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.772411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:26.772850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.772850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:26.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:26.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:26.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:26.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:26.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:26.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:26.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:26.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:26.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:26.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:26.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:26.773111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:26.773574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.773975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:26.773975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.773975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:26.805512 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:26.805679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:26.805849 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:26.805973 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:31:26.807699 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:26.807930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.808566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:26.808566 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.808566 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:26.808566 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:26.808566 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.808566 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:26.808566 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.808566 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:26.809463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.809463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:26.809463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:26.809463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:26.809463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.809463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:26.809463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:26.809463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:26.809463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.809463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:26.809463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:26.809463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:26.809463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:26.809463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:26.809463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:26.809810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:26.809810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:26.809810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.809810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:26.809810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:26.809810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.809810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:26.809810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.809810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:26.810361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.810361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:26.810361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:26.810361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:26.810361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.810361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:26.810361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:26.810361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:31:26.810361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:26.810361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:26.810361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.810361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:26.810628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:26.810628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:26.810628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.810628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:26.810628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:26.810628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:26.810628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:26.810628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:26.810628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:26.810628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:26.810628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:26.810628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.810628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:26.810628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:26.810925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.810925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:26.810925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.810925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:26.811469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.811469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:26.811469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:26.811469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:26.811469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:26.811469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:26.811469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.811469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:26.811469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:26.811469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.811469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:26.811469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:26.811469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.811469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:26.811469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:26.811469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:26.811860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:26.812199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.812199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:26.812199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:26.812199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.812199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:26.812199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.812199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:26.844283 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:26.844870 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:26.847129 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:26.847368 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:26.847998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:26.848334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.849183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:26.849183 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.849183 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:26.849183 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:26.849183 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.849183 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:26.849183 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.849183 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:26.850164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.850164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:26.850164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:26.850164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:26.850164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.850164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:26.850164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:26.850164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:26.850164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.850164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:26.850164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:26.850164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:26.850164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:26.850164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:26.850164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:26.850649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:26.850649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:26.850649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.850649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:26.850649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:26.850649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.850649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:26.850649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.850649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:26.851079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.851079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:26.851079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:26.851079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:26.851079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.851079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:26.851079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:26.851079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:26.851079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:26.851079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.851079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:26.851395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:26.851395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:26.851395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.851395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:26.851395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:26.851395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:26.851395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:26.851395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:26.851395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:26.851395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:26.851395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:26.851395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.851395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:26.851395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:26.852121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.852121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:26.852121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.852121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.852492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:26.853315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:26.853892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.853892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:26.853892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.853892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:26.860584 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:26.860972 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:26.861228 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:26.861509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:26.862044 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:26.862371 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:26.863088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.863763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:26.863763 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.863763 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:26.863763 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:26.863763 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.863763 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:26.863763 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.863763 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:26.864948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.864948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:26.864948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:26.864948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:26.864948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.864948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:26.864948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:26.864948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.864948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:26.864948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:26.864948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:26.864948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.865375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:26.865375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:26.865375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:26.865375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:26.865375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:26.865375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:26.865375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:26.865375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:26.865375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.865375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:26.865375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:26.865375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.865375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:26.865812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.865812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:26.866162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.866162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:26.866162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:26.866162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:26.866162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:26.866162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:26.866162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.866162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:26.866162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:26.866162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.866162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:26.866162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:26.866162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.866162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:26.866162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:26.866162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:26.867016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.867529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:26.867529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.867529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:26.888019 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:26.888019 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:26.888019 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:26.889443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:26.959253 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:26.959253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.959969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:26.959969 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.959969 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:26.959969 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:26.959969 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.959969 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:26.959969 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.959969 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:26.960869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.071 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.960869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:26.960869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:26.960869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:26.960869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.960869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:26.960869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:26.960869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.960869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:26.960869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:26.960869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:26.960869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.961277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:26.961277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:26.961277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:26.961277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:26.961277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:26.961277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:26.961277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:26.961277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:26.961277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.961277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:26.961277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:26.961277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.961277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:26.961526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.961526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:26.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:26.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:26.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:26.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:26.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:26.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:26.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:26.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:26.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:26.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:26.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:26.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:26.962470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:26.963393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:26.963393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:26.963393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:31.767885 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:31.768345 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:31.768386 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:31.769561 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:31.769813 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:31.769960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:31.770078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:31.770215 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:31.770300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:31.770420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:31.770519 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:31.770644 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:31.770831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:31.770880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.771358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:31.771358 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.771358 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:31.771358 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:31.771358 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.771358 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:31.771358 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.771358 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:31.772241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.772241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:31.772241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:31.772241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:31.772241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.772241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:31.772241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:31.772241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.772241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:31.772241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:31.772241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:31.772241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.772489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:31.772489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:31.772489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:31.772489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:31.772489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:31.772489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:31.772489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:31.772489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:31.772489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.772489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:31.772489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:31.772489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.772489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:31.772843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.772843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:31.773023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.773023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:31.773023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:31.773023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:31.773023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:31.773023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:31.773023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.773023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:31.773023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:31.773023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.773023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:31.773023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:31.773023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.773023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:31.773023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:31.773023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:31.773318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.773703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:31.773703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.773703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:31.805945 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:31.806676 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:31.806843 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:31.806968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:31:31.807506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:31.807769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.808662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:31.808662 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.808662 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:31.808662 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:31.808662 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.808662 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:31.808662 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.808662 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:31.809392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.809392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:31.809392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:31.809392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:31.809392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.809392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:31.809392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:31.809392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:31.809392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.809392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:31.809392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:31.809392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:31.809392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:31.809392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:31.809392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:31.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:31.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:31.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:31.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:31.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:31.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.809855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:31.810144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.810144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:31.810144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:31.810144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:31.810144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.810144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:31.810144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:31.810144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:31:31.810144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:31.810144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:31.810144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.810144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:31.810563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:31.810563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:31.810563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.810563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:31.810563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:31.810563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:31.810563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:31.810563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:31.810563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:31.810563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:31.810563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:31.810563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.810563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:31.810563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:31.810936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.810936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:31.810936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.810936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:31.813078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.813078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:31.813078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:31.813078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:31.813078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:31.813078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:31.813078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.813078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:31.813078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:31.813078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.813078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:31.813078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:31.813078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.813078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:31.813078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:31.813078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:31.813435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:31.813868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.813868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:31.813868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:31.813868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.813868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:31.813868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.813868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:31.844611 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:31.845169 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:31.845456 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:31.845664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:31.846120 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:31.846367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.846966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:31.846966 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.846966 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:31.846966 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:31.846966 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.846966 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:31.846966 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.846966 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:31.847904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.847904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:31.847904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:31.847904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:31.847904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.847904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:31.847904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:31.847904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:31.847904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.847904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:31.847904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:31.847904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:31.847904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:31.847904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:31.847904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:31.848365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:31.848365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:31.848365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.848365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:31.848365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:31.848365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.848365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:31.848365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.848365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:31.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:31.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:31.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:31.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:31.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:31.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:31.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:31.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:31.849290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:31.849290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:31.849290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.849290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:31.849290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:31.849290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:31.849290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:31.849290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:31.849290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:31.849290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:31.849290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:31.849290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.849290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:31.849290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:31.849697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.849697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:31.849697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.849697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.852199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:31.852759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:31.853246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.853246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:31.853246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.853246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:31.861007 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:31.861455 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:31.861664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:31.862059 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:31.863556 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:31.863834 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:31.864076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.864701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:31.864701 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.864701 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:31.864701 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:31.864701 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.864701 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:31.864701 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.864701 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:31.865857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.865857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:31.865857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:31.865857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:31.865857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.865857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:31.865857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:31.865857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.865857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:31.865857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:31.865857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:31.865857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.866281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:31.866281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:31.866281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:31.866281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:31.866281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:31.866281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:31.866281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:31.866281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:31.866281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.866281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:31.866281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:31.866281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.866281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:31.866865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.866865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:31.867107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.867107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:31.867107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:31.867107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:31.867107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:31.867107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:31.867107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.867107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:31.867107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:31.867107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.867107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:31.867107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:31.867107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.867107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:31.867107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:31.867107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:31.867774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.868397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:31.868397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.868397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:31.888551 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:31.889060 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:31.889225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:31.890697 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:31.942559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:31.942840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.943302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:31.943302 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.943302 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:31.943302 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:31.943302 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.943302 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:31.943302 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.943302 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:31.944179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.944179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:31.944179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:31.944179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:31.944179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.944179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:31.944179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:31.944179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.944179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:31.944179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:31.944179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:31.944179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.944691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:31.944691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:31.944691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:31.944691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:31.944691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:31.944691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:31.944691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:31.944691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:31.944691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.944691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:31.944691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:31.944691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.944691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:31.945169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.945169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:31.945169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.945169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:31.945169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:31.945169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:31.945169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:31.945169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:31.945169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.945169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:31.945169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:31.945169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.945169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:31.945169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:31.945169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:31.945864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:31.946620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:31.946620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:31.946620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:31.946620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:31.946620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:31.946620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:36.768389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:36.769004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:36.769004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:36.771498 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:36.771749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:36.771886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:36.771997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:36.772133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:36.772273 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:36.772385 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:36.772521 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:36.772680 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:36.772824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:36.772929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.774251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:36.774251 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.774251 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:36.774251 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:36.774251 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.774251 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:36.774251 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.774251 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:36.775473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.775473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:36.775473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:36.775473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:36.775473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.775473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:36.775473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:36.775473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.775473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:36.775473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:36.775473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:36.775473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.775879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:36.775879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:36.775879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:36.775879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:36.775879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:36.775879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:36.775879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:36.775879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:36.775879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.775879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:36.775879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:36.775879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.775879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:36.776129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.776129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:36.778903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.778903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:36.778903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:36.778903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:36.778903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:36.778903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:36.778903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.778903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:36.778903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:36.778903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.778903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:36.778903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:36.778903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.778903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:36.778903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:36.778903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:36.779516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.780008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:36.780008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.780008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:36.806778 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:36.808287 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:36.808549 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:36.808795 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:31:36.809132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:36.809305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.811654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:36.811654 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.811654 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:36.811654 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:36.811654 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.811654 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:36.811654 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.811654 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:36.812456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.812456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:36.812456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:36.812456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:36.812456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.812456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:36.812456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:36.812456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:36.812456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.812456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:36.812456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:36.812456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:36.812456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:36.812456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:36.812456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:36.813030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:36.813030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:36.813030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.813030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:36.813030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:36.813030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.813030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:36.813030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.813030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:36.813505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.813505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:36.813505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:36.813505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:36.813505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.813505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:36.813505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:36.813505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:31:36.813505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:36.813505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:36.813505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.813505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:36.813991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:36.813991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:36.813991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.813991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:36.813991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:36.813991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:36.813991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:36.813991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:36.813991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:36.813991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:36.813991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:36.813991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.813991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:36.813991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:36.814435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.814435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:36.814435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.814435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:36.814817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.814817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:36.814817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:36.814817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:36.814817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:36.814817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:36.814817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.814817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:36.814817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:36.814817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.814817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:36.814817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:36.814817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.814817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:36.814817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:36.814817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:36.815360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:36.816037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.816037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:36.816037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:36.816037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.816037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:36.816037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.816037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:36.845242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:36.845838 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:36.846027 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:36.846204 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:36.846564 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:36.846849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.847577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:36.847577 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.847577 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:36.847577 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:36.847577 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.847577 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:36.847577 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.847577 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:36.850914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.850914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:36.850914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:36.850914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:36.850914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.850914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:36.850914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:36.850914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:36.850914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.850914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:36.850914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:36.850914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:36.850914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:36.850914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:36.850914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:36.851422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:36.851422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:36.851422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.851422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:36.851422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:36.851422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.851422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:36.851422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.851422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:36.852854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.852854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:36.852854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:36.852854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:36.852854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.852854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:36.852854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:36.852854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:36.852854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:36.852854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.852854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:36.853263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:36.853263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:36.853263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.853263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:36.853263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:36.853263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:36.853263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:36.853263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:36.853263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:36.853263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:36.853263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:36.853263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.853263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:36.853263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:36.853673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.853673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:36.853673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.853673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.854136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:36.854631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:36.855862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.855862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:36.855862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.855862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:36.861587 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:36.861918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:36.862086 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:36.862411 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:36.862878 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:36.863190 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:36.863534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.864082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:36.864082 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.864082 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:36.864082 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:36.864082 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.864082 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:36.864082 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.864082 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:36.865110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.865110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:36.865110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:36.865110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:36.865110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.865110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:36.865110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:36.865110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.865110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:36.865110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:36.865110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:36.865110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.865566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:36.865566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:36.865566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:36.865566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:36.865566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:36.865566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:36.865566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:36.865566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:36.865566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.865566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:36.865566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:36.865566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.865566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:36.865954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.865954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:36.866763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.866763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:36.866763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:36.866763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:36.866763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:36.866763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:36.866763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.866763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:36.866763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:36.866763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.866763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:36.866763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:36.866763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.866763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:36.866763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:36.866763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:36.867235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.867816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:36.867816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.867816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:36.889103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:36.890185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:36.890185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:36.890978 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:36.959626 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:36.960390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.961252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:36.961252 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.961252 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:36.961252 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:36.961252 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.961252 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:36.961252 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.961252 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:36.963776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.071 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.963776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:36.963776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:36.963776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:36.963776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.963776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:36.963776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:36.963776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.963776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:36.963776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:36.963776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:36.963776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.964286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:36.964286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:36.964286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:36.964286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:36.964286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:36.964286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:36.964286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:36.964286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:36.964286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.964286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:36.964286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:36.964286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.964286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:36.964659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.964659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:36.964659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.964659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:36.964659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:36.964659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:36.964659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:36.964659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:36.964659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.964659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:36.964659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:36.964659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.964659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:36.964659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:36.964659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:36.965616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:36.966873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:36.966873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:36.966873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:36.966873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:36.966873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:36.966873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:41.769137 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:41.769791 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:41.769791 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:41.772268 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:41.772268 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:41.772268 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:41.772268 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:41.772268 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:41.773242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:41.773242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:41.773242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:41.773242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:41.773242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:41.773918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.775923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:41.775923 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.775923 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:41.775923 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:41.775923 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.775923 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:41.775923 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.775923 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:41.777073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.777073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:41.777073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:41.777073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:41.777073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.777073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:41.777073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:41.777073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.777073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:41.777073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:41.777073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:41.777073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.777912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:41.777912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:41.777912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:41.777912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:41.777912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:41.777912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:41.777912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:41.777912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:41.777912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.777912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:41.777912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:41.777912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.777912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:41.778964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.778964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:41.778964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.778964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:41.778964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:41.778964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:41.778964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:41.778964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:41.778964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.778964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:41.778964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:41.778964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.778964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:41.778964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:41.778964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:41.780019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.781122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:41.781122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:41.781122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.781122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:41.781122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.781122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:41.808326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:41.808764 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:41.808921 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:41.809076 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:31:41.809783 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:41.810101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.811265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:41.811265 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.811265 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:41.811265 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:41.811265 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.811265 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:41.811265 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.811265 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:41.813091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.813091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:41.813091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:41.813091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:41.813091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.813091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:41.813091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:41.813091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:41.813091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.813091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:41.813091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:41.813091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:41.813091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:41.813091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:41.813091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:41.813965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:41.813965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:41.813965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.813965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:41.813965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:41.813965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.813965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:41.813965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.813965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:41.815223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.815223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:41.815223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:41.815223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:41.815223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.815223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:41.815223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:41.815223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:31:41.815223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:41.815223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:41.815223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.815223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:41.815617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:41.815617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:41.815617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.815617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:41.815617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:41.815617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:41.815617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:41.815617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:41.815617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:41.815617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:41.815617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:41.815617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.815617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:41.815617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:41.816016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.816016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:41.816016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.816016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:41.819798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.819798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:41.819798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:41.819798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:41.819798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:41.819798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:41.819798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.819798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:41.819798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:41.819798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.819798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:41.819798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:41.819798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.819798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:41.819798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:41.819798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:41.820570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:41.821005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.821005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:41.821005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:41.821005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.821005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:41.821005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.821005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:41.846782 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:41.847559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:41.847785 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:41.847955 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:41.848992 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:41.849341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.851379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:41.852000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:41.852000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:41.852000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:41.852000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:41.852000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:41.852000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:41.852000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:41.852000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.852000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:41.852000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:41.852000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.852000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:41.852000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.852000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:41.852791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.852791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:41.852791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:41.852791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:41.852791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.852791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:41.852791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:41.852791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:41.852791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:41.852791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.852791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:41.853190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:41.853190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:41.853190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.853190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:41.853190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:41.853190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:41.853190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:41.853190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:41.853190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:41.853190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:41.853190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:41.853190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.853190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:41.853190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:41.853622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.853622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:41.853622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.853622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.855494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:41.856075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:41.856869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.856869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:41.856869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.856869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:41.861925 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:41.862270 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:41.862407 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:41.862630 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:41.863141 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:41.863414 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:41.863769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.864398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:41.864398 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.864398 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:41.864398 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:41.864398 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.864398 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:41.864398 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.864398 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:41.865504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.865504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:41.865504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:41.865504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:41.865504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.865504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:41.865504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:41.865504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.865504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:41.865504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:41.865504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:41.865504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.865823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:41.865823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:41.865823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:41.865823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:41.865823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:41.865823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:41.865823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:41.865823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:41.865823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.865823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:41.865823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:41.865823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.865823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:41.866056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.866056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:41.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:41.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:41.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:41.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:41.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:41.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:41.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:41.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:41.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:41.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:41.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:41.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:41.867887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.868209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:41.868209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.868209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:41.890099 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:41.890319 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:41.890896 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:41.891841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:41.960941 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:41.961217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.961994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:41.961994 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.961994 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:41.961994 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:41.961994 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.961994 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:41.961994 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.961994 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:41.964458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.071 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.964458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:41.964458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:41.964458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:41.964458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.964458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:41.964458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:41.964458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.964458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:41.964458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:41.964458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:41.964458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.964905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:41.964905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:41.964905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:41.964905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:41.964905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:41.964905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:41.964905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:41.964905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:41.964905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.964905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:41.964905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:41.964905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.964905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:41.965115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.965115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:41.965115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.965115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:41.965115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:41.965115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:41.965115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:41.965115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:41.965115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.965115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:41.965115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:41.965115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.965115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:41.965115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:41.965115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:41.965394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:41.966099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:41.966099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:41.966099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:41.966099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:41.966099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:41.966099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:46.769488 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:46.770152 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:46.770152 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:46.771944 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:46.772112 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:46.772241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:46.772384 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:46.772515 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:46.772641 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:46.772733 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:46.772857 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:46.773012 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:46.773012 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:46.773133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.773582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:46.773582 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.773582 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:46.773582 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:46.773582 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.773582 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:46.773582 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.773582 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:46.774429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.774429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:46.774429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:46.774429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:46.774429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.774429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:46.774429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:46.774429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.774429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:46.774429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:46.774429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:46.774429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.774851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:46.774851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:46.774851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:46.774851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:46.774851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:46.774851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:46.774851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:46.774851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:46.774851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.774851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:46.774851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:46.774851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.774851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:46.775172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.775172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:46.775345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.775345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:46.775345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:46.775345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:46.775345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:46.775345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:46.775345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.775345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:46.775345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:46.775345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.775345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:46.775345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:46.775345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.775345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:46.775345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:46.775345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:46.776110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.776629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:46.776629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.776629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:46.808883 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:46.809320 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:46.809509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:46.809705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:31:46.810113 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:46.810344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.810993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:46.810993 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.810993 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:46.810993 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:46.810993 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.810993 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:46.810993 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.810993 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:46.812111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.812111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:46.812111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:46.812111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:46.812111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.812111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:46.812111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:46.812111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:46.812111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.812111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:46.812111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:46.812111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:46.812111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:46.812111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:46.812111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:46.812616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:46.812616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:46.812616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.812616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:46.812616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:46.812616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.812616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:46.812616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.812616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:46.813287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.813287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:46.813287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:46.813287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:46.813287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.813287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:46.813287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:46.813287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:31:46.813287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:46.813287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:46.813287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.813287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:46.813907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:46.813907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:46.813907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.813907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:46.813907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:46.813907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:46.813907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:46.813907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:46.813907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:46.813907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:46.813907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:46.813907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.813907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:46.813907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:46.814234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.814234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:46.814234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.814234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:46.814441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.814441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:46.814441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:46.814441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:46.814441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:46.814441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:46.814441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.814441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:46.814441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:46.814441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.814441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:46.814441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:46.814441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.814441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:46.814441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:46.814441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:46.814955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:46.815305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.815305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:46.815305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:46.815305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.815305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:46.815305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.815305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:46.847488 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:46.847930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:46.848137 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:46.848323 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:46.851938 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:46.852211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.852901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:46.852901 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.852901 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:46.852901 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:46.852901 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.852901 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:46.852901 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.852901 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:46.853692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.853692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:46.853692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:46.853692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:46.853692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.853692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:46.853692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:46.853692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:46.853692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.853692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:46.853692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:46.853692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:46.853692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:46.853692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:46.853692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:46.854011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:46.854011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:46.854011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.854011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:46.854011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:46.854011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.854011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:46.854011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.854011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:46.854726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.854726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:46.854726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:46.854726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:46.854726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.854726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:46.854726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:46.854726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:46.854726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:46.854726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.854726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:46.855010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:46.855010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:46.855010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.855010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:46.855010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:46.855010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:46.855010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:46.855010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:46.855010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:46.855010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:46.855010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:46.855010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.855010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:46.855010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:46.855251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.855251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:46.855251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.855251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.858994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:46.859452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:46.859920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.859920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:46.859920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.859920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:46.862373 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:46.862786 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:46.862965 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:46.863171 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:46.863502 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:46.863793 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:46.864119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.867820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:46.867820 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.867820 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:46.867820 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:46.867820 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.867820 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:46.867820 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.867820 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:46.868801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.868801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:46.868801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:46.868801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:46.868801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.868801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:46.868801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:46.868801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.868801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:46.868801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:46.868801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:46.868801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.869264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:46.869264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:46.869264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:46.869264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:46.869264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:46.869264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:46.869264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:46.869264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:46.869264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.869264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:46.869264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:46.869264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.869264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:46.869826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.869826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:46.869903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.869903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:46.869903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:46.869903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:46.869903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:46.869903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:46.869903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.869903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:46.869903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:46.869903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.869903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:46.869903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:46.869903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.869903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:46.869903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:46.869903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:46.870411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.871034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:46.871034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.871034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:46.890340 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:46.890859 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:46.891005 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:46.891842 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:46.937571 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:46.937905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.938467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:46.938467 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.938467 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:46.938467 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:46.938467 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.938467 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:46.938467 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.938467 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:46.939393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.939393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:46.939393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:46.939393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:46.939393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.939393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:46.939393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:46.939393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.939393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:46.939393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:46.939393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:46.939393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.939883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:46.939883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:46.939883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:46.939883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:46.939883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:46.939883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:46.939883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:46.939883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:46.939883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.939883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:46.939883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:46.939883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.939883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:46.940260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.940260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:46.943612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.943612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:46.943612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:46.943612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:46.943612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:46.943612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:46.943612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.943612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:46.943612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:46.943612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.943612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:46.943612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:46.943612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.943612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:46.943612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:46.943612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:46.943998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:46.944345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:46.944345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:46.944345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:51.770530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:51.771111 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:51.771111 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:51.776020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:51.776314 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:51.776472 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:51.776649 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:51.776807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:51.776953 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:51.777094 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:51.777230 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:51.777368 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:51.777508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:51.777673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.781082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:51.781082 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:51.781082 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:51.781082 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:51.781082 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:51.781082 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:51.781082 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.781082 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:51.781082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.781082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:51.781082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:51.781082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:51.781082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.781082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:51.781082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:51.781856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.781856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:51.781856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:51.781856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:51.781856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.781856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:51.781856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:51.781856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:51.781856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:51.781856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:51.781856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:51.781856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:51.781856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:51.781856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:51.782240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:51.782240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:51.782240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:51.782240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:51.782240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.782240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:51.785602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.785602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:51.785602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:51.785602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:51.785602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:51.785602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:51.785602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.785602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:51.785602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:51.785602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.785602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:51.785602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:51.785602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.785602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:51.785602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:51.785602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:51.786274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:51.787413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:51.787413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.787413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:51.809385 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:51.813722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:51.813722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:51.813722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:31:51.827088 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:51.827324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.833230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:51.833230 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:51.833230 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:51.833230 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:51.833230 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:51.833230 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:51.833230 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.833230 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:51.833510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.833510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:51.833510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:51.833510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:51.833510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.833510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:51.833510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:51.833510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:51.833510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.833510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:51.833510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:51.833510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:51.833510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:51.833510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:51.833510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:51.834054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:51.834054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:51.834054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:51.834054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:51.834054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:51.834054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:51.834054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:51.834054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.834054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:51.837675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.020 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.837675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:51.837675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:51.837675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:51.837675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.837675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:51.837675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:51.837675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:31:51.837675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:51.837675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:51.837675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.837675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:51.838556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:51.838556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:51.838556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.838556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:51.838556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:51.838556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:51.838556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:51.838556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:51.838556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:51.838556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:51.838556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:51.838556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:51.838556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:51.838556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:51.839606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:51.839606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:51.839606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.839606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:51.839606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.839606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:51.839606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:51.839606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:51.839606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:51.839606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:51.839606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.839606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:51.839606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:51.839606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:51.839995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:51.840451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:51.840451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:51.840451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:51.840451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:51.840451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:51.840451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:51.840451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:51.840451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:51.840451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:51.840451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:51.840451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.840451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:51.849055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:51.855716 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:51.855716 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:51.855716 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:51.856326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:51.857521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.860504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:51.860504 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:51.860504 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:51.860504 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:51.860504 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:51.860504 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:51.860504 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.860504 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:51.862370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.862370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:51.862370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:51.862370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:51.862370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.862370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:51.862370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:51.862370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:51.862370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.862370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:51.862370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:51.862370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:51.862370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:51.862370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:51.862370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:51.862783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:51.862783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:51.862783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:51.862783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:51.862783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:51.862783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:51.862783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:51.862783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.862783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:51.865887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.865887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:51.865887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:51.865887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:51.865887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.865887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:51.865887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:51.865887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:51.865887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:51.865887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.865887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:51.866412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:51.866412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:51.866412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.866412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:51.866412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:51.866412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:51.866412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:51.866412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:51.866412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:51.866412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:51.866412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:51.866412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:51.866412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:51.866412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:51.866866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:51.866866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:51.866866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.866866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.880743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:51.883056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:51.883639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:51.883639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:51.883639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.883639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:51.883762 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:51.888165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:51.888165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:51.888165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:51.888165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:51.888707 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:51.889092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.889949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:51.889949 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:51.889949 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:51.889949 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:51.889949 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:51.889949 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:51.889949 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.889949 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:51.895373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.895373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:51.895373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:51.895373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:51.895373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.895373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:51.895373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:51.895373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.895373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:51.895373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:51.895373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:51.895373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.895975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:51.895975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:51.895975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:51.895975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:51.895975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:51.895975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:51.895975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:51.895975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:51.895975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:51.895975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:51.895975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:51.895975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:51.895975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:51.896637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.896637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:51.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:51.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:51.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:51.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:51.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:51.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:51.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:51.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:51.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:51.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:51.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:51.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:51.903300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:51.903866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:51.903866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:51.903866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:51.903966 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:51.907710 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:51.911372 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:51.911372 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:52.019667 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:52.020154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:52.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:52.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:52.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:52.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:52.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:52.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:52.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:52.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:52.022981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.114 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:52.022981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:52.022981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:52.022981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:52.022981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:52.022981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:52.022981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:52.022981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:52.022981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:52.022981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:52.022981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:52.022981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:52.023905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:52.023905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:52.023905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:52.023905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:52.023905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:52.023905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:52.023905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:52.023905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:52.023905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:52.023905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:52.023905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:52.023905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:52.023905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:52.026728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:52.026728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:52.026728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:52.026728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:52.026728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:52.026728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:52.026728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:52.026728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:52.026728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:52.026728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:52.026728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:52.026728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:52.026728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:52.026728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:52.026728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:52.027201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:52.027679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:52.027679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:52.027679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:52.027679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:52.027679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:52.027679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:56.771208 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:56.772996 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:56.773158 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:56.774644 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:56.774947 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:56.775133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:56.775285 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:56.775432 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:56.775601 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:56.775752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:56.775909 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:56.776052 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:56.776191 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:56.776330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.777812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:56.777812 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.777812 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:56.777812 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:56.777812 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.777812 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:56.777812 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.777812 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:56.779281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.779281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:56.779281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:56.779281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:56.779281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.779281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:56.779281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:56.779281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.779281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:56.779281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:56.779281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:56.779281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.779748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:56.779748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:56.779748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:56.779748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:56.779748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:56.779748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:56.779748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:56.779748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:56.779748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.779748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:56.779748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:56.779748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.779748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:56.780353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.780353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:56.780557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.780557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:56.780557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:56.780557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:56.780557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:56.780557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:56.780557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.780557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:56.780557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:56.780557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.780557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:56.780557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:56.780557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.780557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:56.780557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:56.780557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:56.781451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.781910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:56.781910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.781910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:56.816206 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:56.817959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:56.818231 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:56.818339 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:31:56.820099 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:56.820300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.820960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:56.820960 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.820960 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:56.820960 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:56.820960 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.820960 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:56.820960 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.820960 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:56.822632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.822632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:56.822632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:56.822632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:56.822632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.822632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:56.822632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:56.822632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:56.822632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.822632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:56.822632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:56.822632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:56.822632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:56.822632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:56.822632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:56.823223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:56.823223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:56.823223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.823223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:56.823223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:56.823223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.823223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:56.823223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.823223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:56.825442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.825442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:56.825442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:56.825442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:56.825442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.825442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:56.825442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:56.825442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:31:56.825442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:56.825442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:56.825442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.825442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:56.825905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:56.825905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:56.825905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.825905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:56.825905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:56.825905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:56.825905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:56.825905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:56.825905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:56.825905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:56.825905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:56.825905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.825905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:56.825905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:56.826236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.826236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:56.826236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.826236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:56.828932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.828932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:56.828932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:56.828932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:56.828932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:56.828932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:56.828932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.828932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:56.828932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:56.828932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.828932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:56.828932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:56.828932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.828932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:56.828932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:56.828932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:56.829500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:56.829995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.829995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:56.829995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:56.829995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.829995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:56.829995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.829995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:56.854051 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:56.854500 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:56.854694 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:56.854904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:56.855971 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:56.856365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.857101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:56.857101 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.857101 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:56.857101 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:56.857101 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.857101 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:56.857101 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.857101 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:56.857909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.857909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:31:56.857909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:56.857909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:56.857909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.857909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:56.857909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:56.857909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:31:56.857909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.857909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:31:56.857909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:56.857909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:31:56.857909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:56.857909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:31:56.857909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:31:56.858401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:56.858401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:31:56.858401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.858401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:31:56.858401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:31:56.858401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.858401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:31:56.858401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.858401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:31:56.858694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.858694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:56.858694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:56.858694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:56.858694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.858694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:56.858694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:56.858694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:56.858694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:56.858694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.858694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:56.859123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:56.859123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:56.859123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.859123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:56.859123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:56.859123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:56.859123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:56.859123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:56.859123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:56.859123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:56.859123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:56.859123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.859123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:56.859123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:56.859519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.859519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:56.859519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.859519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:56.859519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.859519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:56.859519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:56.859519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:56.859519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:56.859519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:56.859519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.859519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:56.859519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:56.859519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:56.860196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:56.860781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:56.860781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:56.860781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:56.860781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:56.860781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.860781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:56.860781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:56.860781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.860781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:56.860781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.860781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:56.887159 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:56.889216 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:56.889322 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:56.889496 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:56.890025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:56.890328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:56.890710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.891389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:56.891389 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.891389 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:56.891389 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:56.891389 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.891389 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:56.891389 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.891389 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:56.892321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.892321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:56.892321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:56.892321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:56.892321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.892321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:56.892321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:56.892321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.892321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:56.892321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:56.892321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:56.892321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.892594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:56.892594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:56.892594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:56.892594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:56.892594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:56.892594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:56.892594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:56.892594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:56.892594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.892594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:56.892594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:56.892594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.892594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:56.892814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.892814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:56.893245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.893245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:56.893245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:56.893245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:56.893245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:56.893245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:56.893245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.893245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:56.893245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:56.893245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.893245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:56.893245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:56.893245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.893245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:56.893245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:56.893245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:56.893576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.893897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:56.893897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.893897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:31:56.907820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:31:56.908219 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:31:56.908379 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:31:56.909189 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:31:56.962289 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:31:56.962899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.963169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:31:56.963169 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.963169 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:31:56.963169 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:31:56.963169 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.963169 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:31:56.963169 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.963169 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:31:56.964069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.964069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:31:56.964069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:56.964069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:31:56.964069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.964069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:56.964069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:56.964069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.964069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:56.964069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:56.964069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:31:56.964069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.964627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:31:56.964627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:56.964627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:31:56.964627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:56.964627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:31:56.964627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:31:56.964627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:56.964627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:31:56.964627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.964627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:31:56.964627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:31:56.964627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.964627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:31:56.965179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.965179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:31:56.965327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.965327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:31:56.965327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:31:56.965327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:31:56.965327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:31:56.965327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:31:56.965327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.965327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:31:56.965327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:31:56.965327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.965327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:31:56.965327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:31:56.965327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.965327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:31:56.965327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:31:56.965327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:31:56.965737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:31:56.966191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:31:56.966191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:31:56.966191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:01.773025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:01.773970 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:01.775117 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:01.776465 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:01.776697 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:01.776892 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:01.777023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:01.777099 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:01.777212 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:01.777599 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:01.777722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:01.777845 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:01.777951 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:01.778063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.782087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:01.782087 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.782087 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:01.782087 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:01.782087 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.782087 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:01.782087 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.782087 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:01.785273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.785273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:01.785273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:01.785273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:01.785273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.785273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:01.785273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:01.785273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.785273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:01.785273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:01.785273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:01.785273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.785677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:01.785677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:01.785677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:01.785677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:01.785677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:01.785677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:01.785677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:01.785677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:01.785677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.785677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:01.785677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:01.785677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.785677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:01.785945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.785945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:01.787105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.787105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:01.787105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:01.787105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:01.787105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:01.787105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:01.787105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.787105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:01.787105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:01.787105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.787105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:01.787105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:01.787105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.787105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:01.787105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:01.787105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:01.787419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.787776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:01.787776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.787776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:01.817959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:01.818365 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:01.818933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:01.818933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:32:01.819263 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:01.819486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.820057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:01.820057 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.820057 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:01.820057 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:01.820057 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.820057 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:01.820057 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.820057 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:01.821011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.821011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:01.821011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:01.821011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:01.821011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.821011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:01.821011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:01.821011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:01.821011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.821011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:01.821011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:01.821011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:01.821011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:01.821011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:01.821011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:01.821378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:01.821378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:01.821378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.821378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:01.821378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:01.821378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.821378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:01.821378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.821378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:01.822628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.822628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:01.822628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:01.822628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:01.822628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.822628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:01.822628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:01.822628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:32:01.822628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:01.822628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:01.822628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.822628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:01.823048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:01.823048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:01.823048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.823048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:01.823048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:01.823048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:01.823048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:01.823048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:01.823048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:01.823048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:01.823048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:01.823048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.823048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:01.823048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:01.823343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.823343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:01.823343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.823343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:01.823343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.823343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:01.823343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:01.823343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:01.823343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:01.823343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:01.823343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.823343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:01.823343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:01.823343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:01.823671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:01.824130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:01.824130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:01.824130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:01.824130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:01.824130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:01.824130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.824130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:01.824130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:01.824130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.824130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:01.824130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.824130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:01.855157 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:01.855764 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:01.857664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:01.857664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:01.857664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:01.857664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.857664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:01.857664 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.857664 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:01.857664 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:01.857664 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.857664 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:01.857664 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.857664 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:01.859120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.859120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:01.859120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:01.859120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:01.859120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.859120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:01.859120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:01.859120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:01.859120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.859120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:01.859120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:01.859120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:01.859120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:01.859120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:01.859120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:01.859524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:01.859524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:01.859524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.859524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:01.859524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:01.859524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.859524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:01.859524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.859524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:01.859955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.859955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:01.859955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:01.859955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:01.859955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.859955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:01.859955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:01.859955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:01.859955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:01.859955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.859955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:01.860256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:01.860256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:01.860256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.860256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:01.860256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:01.860256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:01.860256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:01.860256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:01.860256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:01.860256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:01.860256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:01.860256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.860256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:01.860256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:01.860630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.860630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:01.860630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.860630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.862207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:01.862745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:01.863377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.863377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:01.863377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.863377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:01.890602 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:01.890602 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:01.890602 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:01.890602 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:01.890602 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:01.890602 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:01.890955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.893853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:01.893853 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.893853 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:01.893853 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:01.893853 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.893853 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:01.893853 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.893853 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:01.893853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.893853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:01.893853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:01.893853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:01.893853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.893853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:01.893853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:01.894587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.894587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:01.894587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:01.894587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:01.894587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.894587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:01.894587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:01.894587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:01.894587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:01.894587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:01.894587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:01.894587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:01.894587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:01.894587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.895001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:01.895001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:01.895001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.895001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:01.895001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.895001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:01.895001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.895001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:01.895001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:01.895001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:01.895001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:01.895001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:01.895001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.895001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:01.895425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:01.895999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:01.895999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:01.895999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:01.895999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.895999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:01.895999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:01.895999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.895999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:01.895999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.895999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:01.911505 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:01.912004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:01.912274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:01.913894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:01.980322 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:01.980659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.981400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:01.981400 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.981400 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:01.981400 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:01.981400 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.981400 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:01.981400 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.981400 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:01.982885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.069 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.982885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:01.982885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:01.982885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:01.982885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.982885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:01.982885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:01.982885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.982885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:01.982885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:01.982885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:01.982885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:01.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:01.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:01.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:01.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:01.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:01.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:01.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:01.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:01.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:01.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:01.983660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.983660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:01.983922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.983922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:01.983922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:01.983922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:01.983922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:01.983922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:01.983922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.983922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:01.983922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:01.983922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.983922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:01.983922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:01.983922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.983922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:01.983922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:01.983922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:01.984350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:01.984909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:01.984909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:01.984909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:06.774201 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:06.775474 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:06.775474 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:06.776822 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:06.777157 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:06.777416 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:06.777601 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:06.777798 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:06.777953 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:06.778109 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:06.778249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:06.778363 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:06.778496 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:06.778643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.781432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:06.781432 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.781432 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:06.781432 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:06.781432 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.781432 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:06.781432 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.781432 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:06.782488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.782488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:06.782488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:06.782488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:06.782488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.782488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:06.782488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:06.782488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.782488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:06.782488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:06.782488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:06.782488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.783303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:06.783303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:06.783303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:06.783303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:06.783303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:06.783303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:06.783303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:06.783303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:06.783303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.783303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:06.783303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:06.783303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.783303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:06.783739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.783739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:06.783739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.783739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:06.783739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:06.783739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:06.783739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:06.783739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:06.783739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.783739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:06.783739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:06.783739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.783739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:06.783739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:06.783739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:06.784293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.784941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:06.784941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:06.784941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.784941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:06.784941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.784941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:06.818569 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:06.821475 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:06.821633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:06.821769 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:32:06.822126 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:06.822336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.822847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:06.822847 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.822847 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:06.822847 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:06.822847 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.822847 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:06.822847 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.822847 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:06.824581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.824581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:06.824581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:06.824581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:06.824581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.824581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:06.824581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:06.824581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:06.824581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.824581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:06.824581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:06.824581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:06.824581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:06.824581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:06.824581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:06.824981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:06.824981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:06.824981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.824981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:06.824981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:06.824981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.824981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:06.824981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.824981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:06.825439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.825439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:06.825439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:06.825439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:06.825439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.825439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:06.825439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:06.825439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:32:06.825439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:06.825439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:06.825439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.825439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:06.825729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:06.825729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:06.825729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.825729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:06.825729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:06.825729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:06.825729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:06.825729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:06.825729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:06.825729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:06.825729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:06.825729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.825729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:06.825729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:06.825995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.825995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:06.825995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.825995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:06.826474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.826474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:06.826474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:06.826474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:06.826474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:06.826474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:06.826474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.826474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:06.826474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:06.826474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.826474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:06.826474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:06.826474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.826474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:06.826474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:06.826474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:06.826846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:06.827244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.827244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:06.827244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:06.827244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.827244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:06.827244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.827244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:06.856913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:06.857352 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:06.857501 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:06.857668 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:06.858168 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:06.858478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.859166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:06.859166 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.859166 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:06.859166 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:06.859166 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.859166 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:06.859166 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.859166 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:06.859889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.859889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:06.859889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:06.859889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:06.859889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.859889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:06.859889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:06.859889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:06.859889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.859889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:06.859889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:06.859889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:06.859889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:06.859889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:06.859889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:06.860136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:06.860136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:06.860136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.860136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:06.860136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:06.860136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.860136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:06.860136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.860136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:06.860789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.860789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:06.860789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:06.860789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:06.860789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.860789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:06.860789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:06.860789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:06.860789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:06.860789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.860789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:06.861175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:06.861175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:06.861175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.861175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:06.861175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:06.861175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:06.861175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:06.861175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:06.861175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:06.861175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:06.861175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:06.861175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.861175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:06.861175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:06.861560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.861560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:06.861560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.861560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.861873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:06.862471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:06.863038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.863038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:06.863038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.863038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:06.889851 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:06.890272 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:06.890423 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:06.891103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:06.891482 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:06.891752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:06.892072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.892671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:06.892671 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.892671 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:06.892671 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:06.892671 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.892671 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:06.892671 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.892671 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:06.893360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.893360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:06.893360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:06.893360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:06.893360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.893360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:06.893360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:06.893360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.893360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:06.893360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:06.893360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:06.893360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.893805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:06.893805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:06.893805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:06.893805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:06.893805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:06.893805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:06.893805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:06.893805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:06.893805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.893805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:06.893805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:06.893805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.893805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:06.894160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.894160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:06.894305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.894305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:06.894305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:06.894305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:06.894305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:06.894305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:06.894305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.894305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:06.894305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:06.894305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.894305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:06.894305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:06.894305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.894305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:06.894305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:06.894305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:06.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.895314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:06.895314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.895314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:06.912181 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:06.912585 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:06.914080 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:06.914749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:06.971313 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:06.971313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.971313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:06.971313 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.971313 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:06.971313 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:06.971313 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.971313 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:06.971313 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.971313 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:06.972478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.972478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:06.972478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:06.972478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:06.972478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.972478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:06.972478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:06.972478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.972478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:06.972478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:06.972478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:06.972478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.972868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:06.972868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:06.972868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:06.972868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:06.972868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:06.972868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:06.972868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:06.972868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:06.972868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.972868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:06.972868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:06.972868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.972868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:06.973257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.973257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:06.973597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.973597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:06.973597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:06.973597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:06.973597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:06.973597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:06.973597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.973597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:06.973597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:06.973597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.973597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:06.973597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:06.973597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.973597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:06.973597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:06.973597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:06.974166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:06.974746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:06.974746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:06.974746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:11.776815 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:11.777348 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:11.777484 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:11.781563 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:11.781563 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:11.781563 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:11.781563 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:11.781563 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:11.781841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:11.781841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:11.781841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:11.784006 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:11.784006 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:11.784006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.784163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:11.784163 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.784163 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:11.784163 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:11.784163 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.784163 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:11.784163 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.784163 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:11.785286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.785286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:11.785286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:11.785286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:11.785286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.785286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:11.785286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:11.785286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.785286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:11.785286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:11.785286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:11.785286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.785741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:11.785741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:11.785741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:11.785741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:11.785741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:11.785741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:11.785741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:11.785741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:11.785741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.785741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:11.785741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:11.785741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.785741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:11.786116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.786116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:11.786493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.786493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:11.786493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:11.786493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:11.786493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:11.786493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:11.786493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.786493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:11.786493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:11.786493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.786493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:11.786493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:11.786493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.786493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:11.786493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:11.786493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:11.787819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.788362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:11.788362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.788362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:11.821576 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:11.822400 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:11.822639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:11.822823 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:32:11.823331 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:11.823695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.827165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:11.827165 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.827165 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:11.827165 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:11.827165 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.827165 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:11.827165 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.827165 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:11.828007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.828007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:11.828007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:11.828007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:11.828007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.828007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:11.828007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:11.828007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:11.828007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.828007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:11.828007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:11.828007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:11.828007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:11.828007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:11.828007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:11.828579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:11.828579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:11.828579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.828579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:11.828579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:11.828579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.828579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:11.828579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.828579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:11.829091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.829091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:11.829091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:11.829091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:11.829091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.829091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:11.829091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:11.829091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:32:11.829091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:11.829091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:11.829091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.829091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:11.829433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:11.829433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:11.829433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.829433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:11.829433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:11.829433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:11.829433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:11.829433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:11.829433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:11.829433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:11.829433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:11.829433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.829433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:11.829433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:11.829839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.829839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:11.829839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.829839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:11.833220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.833220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:11.833220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:11.833220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:11.833220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:11.833220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:11.833220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.833220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:11.833220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:11.833220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.833220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:11.833220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:11.833220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.833220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:11.833220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:11.833220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:11.833838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:11.834600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.834600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:11.834600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:11.834600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.834600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:11.834600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.834600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:11.857366 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:11.858248 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:11.858248 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:11.858248 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:11.859196 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:11.860214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.863424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:11.864006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:11.864006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:11.864006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:11.864006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:11.864006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:11.864006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:11.864006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:11.864006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.864006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:11.864006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:11.864006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.864006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:11.864006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.864006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:11.864006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.864332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:11.864332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:11.864332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:11.864332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.864332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:11.864332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:11.864332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:11.864332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:11.864332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.864332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:11.864332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:11.864332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:11.864332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.864630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:11.864630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:11.864630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:11.864630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:11.864630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:11.864630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:11.864630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:11.864630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:11.864630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.864630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:11.864630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:11.864630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.864630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:11.864863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.864863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:11.864863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.864863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:11.864863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:11.864863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:11.864863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:11.864863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:11.864863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.864863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:11.864863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:11.864863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.864863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:11.864863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:11.864863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:11.865244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:11.865588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:11.865588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.865588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:11.865588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:11.865588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.865588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:11.865588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.865588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:11.891515 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:11.894562 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:11.896139 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:11.896583 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:11.897276 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:11.897518 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:11.897828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.898591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:11.898591 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.898591 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:11.898591 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:11.898591 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.898591 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:11.898591 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.898591 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:11.899442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.899442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:11.899442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:11.899442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:11.899442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.899442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:11.899442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:11.899442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.899442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:11.899442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:11.899442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:11.899442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.899891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:11.899891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:11.899891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:11.899891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:11.899891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:11.899891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:11.899891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:11.899891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:11.899891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.899891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:11.899891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:11.899891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.899891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:11.900411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.900411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:11.900411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.900411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:11.900411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:11.900411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:11.900411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:11.900411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:11.900411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.900411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:11.900411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:11.900411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.900411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:11.900411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:11.900411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:11.900895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.901448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:11.901448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:11.901448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.901448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:11.901448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.901448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:11.913313 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:11.913719 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:11.914414 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:11.915412 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:11.976092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:11.976324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.978289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:11.978289 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.978289 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:11.978289 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:11.978289 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.978289 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:11.978289 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.978289 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:11.979498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.064 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.979498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:11.979498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:11.979498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:11.979498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.979498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:11.979498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:11.979498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.979498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:11.979498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:11.979498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:11.979498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.979926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:11.979926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:11.979926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:11.979926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:11.979926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:11.979926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:11.979926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:11.979926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:11.979926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.979926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:11.979926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:11.979926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.979926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:11.980288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.980288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:11.981168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.981168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:11.981168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:11.981168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:11.981168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:11.981168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:11.981168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.981168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:11.981168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:11.981168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.981168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:11.981168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:11.981168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.981168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:11.981168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:11.981168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:11.981643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:11.982161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:11.982161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:11.982161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:16.779876 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:16.780420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:16.780420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:16.781935 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:16.782437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:16.782732 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:16.782988 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:16.783192 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:16.783812 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:16.783812 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:16.783812 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:16.783812 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:16.784034 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:16.784199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.784891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:16.784891 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.784891 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:16.784891 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:16.784891 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.784891 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:16.784891 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.784891 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:16.785821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.785821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:16.785821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:16.785821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:16.785821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.785821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:16.785821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:16.785821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.785821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:16.785821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:16.785821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:16.785821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.786320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:16.786320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:16.786320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:16.786320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:16.786320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:16.786320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:16.786320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:16.786320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:16.786320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.786320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:16.786320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:16.786320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.786320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:16.786860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.786860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:16.787568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.787568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:16.787568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:16.787568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:16.787568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:16.787568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:16.787568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.787568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:16.787568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:16.787568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.787568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:16.787568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:16.787568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.787568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:16.787568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:16.787568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:16.788215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.788922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:16.788922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.788922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:16.823045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:16.823045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:16.823045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:16.823366 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:32:16.824233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:16.824828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.826103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:16.826103 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.826103 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:16.826103 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:16.826103 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.826103 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:16.826103 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.826103 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:16.826738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.826738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:16.826738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:16.826738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:16.826738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.826738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:16.826738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:16.826738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:16.826738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.826738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:16.826738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:16.826738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:16.826738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:16.826738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:16.826738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:16.827277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:16.827277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:16.827277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.827277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:16.827277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:16.827277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.827277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:16.827277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.827277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:16.827650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.827650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:16.827650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:16.827650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:16.827650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.827650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:16.827650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:16.827650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:32:16.827650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:16.827650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:16.827650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.827650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:16.828089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:16.828089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:16.828089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.828089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:16.828089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:16.828089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:16.828089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:16.828089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:16.828089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:16.828089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:16.828089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:16.828089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.828089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:16.828089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:16.828716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.828716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:16.828716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.828716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:16.828861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.828861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:16.828861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:16.828861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:16.828861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:16.828861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:16.828861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.828861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:16.828861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:16.828861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.828861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:16.828861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:16.828861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.828861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:16.828861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:16.828861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:16.829527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:16.831179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.831179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:16.831179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:16.831179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.831179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:16.831179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.831179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:16.858165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:16.858963 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:16.858963 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:16.859232 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:16.859992 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:16.860706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.861717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:16.861717 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.861717 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:16.861717 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:16.861717 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.861717 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:16.861717 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.861717 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:16.862891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.862891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:16.862891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:16.862891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:16.862891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.862891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:16.862891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:16.862891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:16.862891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.862891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:16.862891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:16.862891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:16.862891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:16.862891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:16.862891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:16.863318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:16.863318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:16.863318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.863318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:16.863318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:16.863318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.863318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:16.863318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.863318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:16.864262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.864262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:16.864262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:16.864262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:16.864262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.864262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:16.864262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:16.864262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:16.864262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:16.864262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.864262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:16.864760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:16.864760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:16.864760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.864760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:16.864760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:16.864760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:16.864760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:16.864760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:16.864760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:16.864760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:16.864760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:16.864760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.864760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:16.864760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:16.865130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.865130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:16.865130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.865130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.866267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:16.866806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:16.867263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.867263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:16.867263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.867263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:16.894894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:16.898063 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:16.898236 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:16.898346 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:16.899184 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:16.899876 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:16.900335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.900936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:16.900936 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.900936 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:16.900936 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:16.900936 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.900936 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:16.900936 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.900936 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:16.901836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.901836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:16.901836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:16.901836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:16.901836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.901836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:16.901836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:16.901836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.901836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:16.901836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:16.901836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:16.901836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.902145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:16.902145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:16.902145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:16.902145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:16.902145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:16.902145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:16.902145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:16.902145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:16.902145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.902145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:16.902145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:16.902145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.902145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:16.902369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.902369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:16.902956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.902956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:16.902956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:16.902956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:16.902956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:16.902956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:16.902956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.902956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:16.902956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:16.902956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.902956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:16.902956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:16.902956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.902956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:16.902956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:16.902956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:16.903295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.903825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:16.903825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.903825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:16.914272 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:16.915872 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:16.916069 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:16.916869 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:16.989652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:16.990092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.991017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:16.991017 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.991017 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:16.991017 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:16.991017 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.991017 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:16.991017 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.991017 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:16.991969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.075 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.991969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:16.991969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:16.991969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:16.991969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.991969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:16.991969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:16.991969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.991969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:16.991969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:16.991969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:16.991969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.992377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:16.992377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:16.992377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:16.992377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:16.992377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:16.992377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:16.992377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:16.992377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:16.992377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.992377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:16.992377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:16.992377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.992377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:16.992853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.992853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:16.996076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.996076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:16.996076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:16.996076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:16.996076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:16.996076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:16.996076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.996076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:16.996076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:16.996076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.996076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:16.996076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:16.996076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.996076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:16.996076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:16.996076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:16.996654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:16.997061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:16.997061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:16.997061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:21.780446 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:21.782724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:21.782724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:21.785856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:21.785856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:21.785856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:21.785856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:21.785856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:21.786178 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:21.787705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:21.787705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:21.788573 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:21.789561 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:21.789828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.793273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:21.793273 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.793273 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:21.793273 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:21.793273 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.793273 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:21.793273 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.793273 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:21.793273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.793273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:21.793273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:21.793273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:21.793273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.793273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:21.793273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:21.794031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.794031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:21.794031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:21.794031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:21.794031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.794031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:21.794031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:21.794031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:21.794031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:21.794031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:21.794031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:21.794031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:21.794031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:21.794031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.794412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:21.794412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:21.794412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.794412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:21.794412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.794412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:21.794412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.794412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:21.794412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:21.794412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:21.794412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:21.794412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:21.794412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.794412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:21.795011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:21.795675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:21.795675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:21.795675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:21.795675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.795675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:21.795675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:21.795675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.795675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:21.795675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.795675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:21.823093 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:21.824748 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:21.824748 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:21.824748 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:32:21.825775 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:21.825775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.826362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:21.826362 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.826362 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:21.826362 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:21.826362 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.826362 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:21.826362 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.826362 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:21.828433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.828433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:21.828433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:21.828433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:21.828433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.828433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:21.828433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:21.828433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:21.828433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.828433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:21.828433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:21.828433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:21.828433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:21.828433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:21.828433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:21.828960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:21.828960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:21.828960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.828960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:21.828960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:21.828960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.828960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:21.828960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.828960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:21.830801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.830801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:21.830801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:21.830801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:21.830801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.830801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:21.830801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:21.830801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:32:21.830801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:21.830801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:21.830801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.830801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:21.831384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:21.831384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:21.831384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.831384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:21.831384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:21.831384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:21.831384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:21.831384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:21.831384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:21.831384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:21.831384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:21.831384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.831384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:21.831384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:21.831935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.831935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:21.831935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.831935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:21.833635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.833635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:21.833635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:21.833635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:21.833635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:21.833635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:21.833635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.833635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:21.833635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:21.833635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.833635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:21.833635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:21.833635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.833635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:21.833635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:21.833635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:21.834143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:21.834725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.834725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:21.834725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:21.834725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.834725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:21.834725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.834725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:21.859171 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:21.860497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:21.860497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:21.860497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:21.861186 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:21.861437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.862327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:21.862327 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.862327 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:21.862327 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:21.862327 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.862327 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:21.862327 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.862327 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:21.865176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.865176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:21.865176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:21.865176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:21.865176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.865176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:21.865176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:21.865176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:21.865176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.865176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:21.865176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:21.865176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:21.865176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:21.865176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:21.865176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:21.868576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:21.868576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:21.868576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.868576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:21.868576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:21.868576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.868576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:21.868576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.868576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:21.870924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.870924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:21.870924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:21.870924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:21.870924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.870924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:21.870924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:21.870924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:21.870924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:21.870924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.870924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:21.871351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:21.871351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:21.871351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.871351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:21.871351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:21.871351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:21.871351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:21.871351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:21.871351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:21.871351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:21.871351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:21.871351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.871351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:21.871351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:21.871651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.871651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:21.871651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.871651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:21.871651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.871651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:21.871651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:21.871651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:21.871651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:21.871651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:21.871651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.871651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:21.871651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:21.871651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:21.872058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:21.872496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:21.872496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:21.872496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:21.872496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:21.872496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.872496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:21.872496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:21.872496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.872496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:21.872496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.872496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:21.898099 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:21.899850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:21.899850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:21.899850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:21.900426 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:21.900730 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:21.901065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.903182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:21.903182 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.903182 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:21.903182 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:21.903182 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.903182 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:21.903182 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.903182 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:21.903182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.903182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:21.903182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:21.903182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:21.903182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.903182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:21.903182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:21.903953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.903953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:21.903953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:21.903953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:21.903953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.903953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:21.903953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:21.903953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:21.903953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:21.903953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:21.903953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:21.903953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:21.903953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:21.903953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.904512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:21.904512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:21.904512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.904512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:21.904512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.904512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:21.906107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.906107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:21.906107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:21.906107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:21.906107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:21.906107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:21.906107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.906107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:21.906107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:21.906107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.906107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:21.906107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:21.906107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.906107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:21.906107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:21.906107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:21.907463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.908144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:21.908144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.908144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:21.916023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:21.918328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:21.918328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:21.918328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:21.935470 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=233 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:32:21.977802 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:21.978200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:21.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:21.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:21.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:21.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:21.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.062 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:21.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:21.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:21.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:21.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:21.982072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.982072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:21.982072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:21.982072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:21.982072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.982072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:21.982072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:21.982072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:21.982072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:21.982072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:21.982072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:21.982072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:21.982072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:21.982072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.982454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:21.982454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:21.982454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.982454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:21.982454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.982454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:21.982454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.982454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:21.982454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:21.982454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:21.982454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:21.982454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:21.982454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.982454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:21.982912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:21.983745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:21.983745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:21.983745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:21.983745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:21.983745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:21.983745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:21.983745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:21.983745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:21.983745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:26.781513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:26.782297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:26.782297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:26.783109 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:26.783287 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-b4d3d35b-2dc5-47d7-8071-6532ef9d4a2f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:26.783438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:26.783608 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:26.783754 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:26.783901 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:26.784059 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:26.784200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:26.784294 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:26.784393 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:26.784594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.784989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:26.784989 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.784989 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:26.784989 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:26.784989 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.784989 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:26.784989 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.784989 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:26.785926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.785926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:26.785926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:26.785926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:26.785926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.785926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:26.785926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:26.785926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.785926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:26.785926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:26.785926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:26.785926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.786701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:26.786701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:26.786701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:26.786701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:26.786701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:26.786701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:26.786701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:26.786701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:26.786701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.786701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:26.786701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:26.786701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.786701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:26.787128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.787128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:26.787128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.787128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:26.787128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:26.787128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:26.787128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:26.787128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:26.787128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.787128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:26.787128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:26.787128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.787128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:26.787128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:26.787128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:26.787415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.787809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:26.787809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:26.787809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.787809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:26.787809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.787809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:26.826565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:26.826565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:26.826565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:26.826565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:32:26.827981 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:26.828175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.828706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:26.828706 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.828706 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:26.828706 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:26.828706 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.828706 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:26.828706 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.828706 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:26.829436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.829436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:26.829436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:26.829436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:26.829436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.829436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:26.829436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:26.829436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:26.829436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.829436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:26.829436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:26.829436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:26.829436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:26.829436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:26.829436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:26.830320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:26.830320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:26.830320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.830320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:26.830320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:26.830320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.830320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:26.830320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.830320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:26.830320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.830320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:26.830320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:26.830320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:26.830320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.831102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:26.831102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:26.831102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:32:26.831102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:26.831102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:26.831102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.831102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:26.831102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:26.831102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:26.831102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.831102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:26.831102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:26.831102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:26.831960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:26.831960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:26.831960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:26.831960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:26.831960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:26.831960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.831960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:26.831960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:26.831960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.831960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:26.831960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.831960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:26.832558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.832558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:26.832558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:26.832558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:26.832558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:26.832558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:26.832558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.832558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:26.832558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:26.832558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.832558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:26.832558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:26.832558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.832558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:26.832558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:26.832558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:26.833471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:26.835791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.835791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:26.835791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:26.835791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.835791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:26.835791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.835791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:26.859778 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:26.860699 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:26.860895 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:26.860997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:26.861251 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:26.861445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.861939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:26.861939 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.861939 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:26.861939 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:26.861939 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.861939 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:26.861939 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.861939 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:26.862490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.862490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:26.862490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:26.862490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:26.862490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.862490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:26.862490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:26.862490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:26.862490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.862490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:26.862490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:26.862490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:26.862490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:26.862490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:26.862490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:26.863795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:26.863795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:26.863795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.863795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:26.863795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:26.863795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.863795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:26.863795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.863795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:26.863795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.863795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:26.863795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:26.863795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:26.863795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.864113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:26.864113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:26.864113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:26.864113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:26.864113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.864113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:26.864113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:26.864113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:26.864113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.864113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:26.864113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:26.864113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:26.864113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:26.864432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:26.864432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:26.864432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:26.864432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:26.864432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.864432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:26.864432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:26.864432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.864432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:26.864432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.864432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:26.864432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:26.864691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.865177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:26.898669 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:26.899120 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:26.899254 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:26.899451 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:26.899971 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:26.902835 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:26.903261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.903979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:26.903979 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.903979 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:26.903979 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:26.903979 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.903979 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:26.903979 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.903979 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:26.904981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.904981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:26.904981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:26.904981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:26.904981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.904981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:26.904981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:26.904981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.904981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:26.904981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:26.904981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:26.904981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.905437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:26.905437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:26.905437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:26.905437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:26.905437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:26.905437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:26.905437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:26.905437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:26.905437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.905437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:26.905437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:26.905437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.905437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:26.905957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.905957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:26.906165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.906165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:26.906165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:26.906165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:26.906165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:26.906165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:26.906165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.906165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:26.906165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:26.906165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.906165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:26.906165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:26.906165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.906165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:26.906165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:26.906165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:26.906704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.907445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:26.907445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.907445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:26.916556 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:26.918900 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:26.919077 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:26.919803 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:26.989699 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:26.989699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.989916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:26.989916 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.989916 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:26.989916 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:26.989916 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.989916 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:26.989916 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.989916 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:26.990913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.071 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.990913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:26.990913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:26.990913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:26.990913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.990913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:26.990913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:26.990913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.990913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:26.990913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:26.990913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:26.990913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.991190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:26.991190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:26.991190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:26.991190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:26.991190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:26.991190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:26.991190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:26.991190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:26.991190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.991190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:26.991190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:26.991190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.991190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:26.991424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.991424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:26.992059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.992059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:26.992059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:26.992059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:26.992059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:26.992059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:26.992059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.992059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:26.992059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:26.992059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.992059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:26.992059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:26.992059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.992059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:26.992059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:26.992059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:26.992495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:26.992850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:26.992850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:26.992850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:31.782256 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:31.784966 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:31.785589 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:31.786910 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:31.787447 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:31.787447 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:31.787447 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:31.787769 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:31.788001 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:31.788290 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:31.788442 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:31.788701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:31.788922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.790008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:31.790008 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.790008 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:31.790008 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:31.790008 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.790008 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:31.790008 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.790008 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:31.791909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.791909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:31.791909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:31.791909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:31.791909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.791909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:31.791909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:31.791909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.791909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:31.791909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:31.791909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:31.791909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.792350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:31.792350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:31.792350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:31.792350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:31.792350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:31.792350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:31.792350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:31.792350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:31.792350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.792350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:31.792350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:31.792350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.792350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:31.792713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.792713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:31.794060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.794060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:31.794060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:31.794060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:31.794060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:31.794060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:31.794060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.794060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:31.794060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:31.794060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.794060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:31.794060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:31.794060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.794060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:31.794060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:31.794060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:31.794493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.796129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:31.796129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.796129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:31.824800 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:31.825215 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:31.825378 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:31.825556 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:32:31.826081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:31.826466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.827175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:31.827175 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.827175 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:31.827175 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:31.827175 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.827175 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:31.827175 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.827175 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:31.828223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.828223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:31.828223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:31.828223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:31.828223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.828223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:31.828223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:31.828223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:31.828223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.828223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:31.828223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:31.828223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:31.828223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:31.828223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:31.828223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:31.828516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:31.828516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:31.828516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.828516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:31.828516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:31.828516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.828516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:31.828516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.828516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:31.829461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.829461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:31.829461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:31.829461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:31.829461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.829461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:31.829461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:31.829461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:32:31.829461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:31.829461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:31.829461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.829461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:31.829890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:31.829890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:31.829890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.829890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:31.829890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:31.829890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:31.829890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:31.829890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:31.829890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:31.829890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:31.829890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:31.829890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.829890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:31.829890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:31.830160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.830160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:31.830160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.830160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:31.831199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.831199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:31.831199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:31.831199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:31.831199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:31.831199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:31.831199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.831199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:31.831199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:31.831199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.831199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:31.831199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:31.831199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.831199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:31.831199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:31.831199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:31.831522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:31.831939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.831939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:31.831939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:31.831939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.831939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:31.831939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.831939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:31.861179 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:31.861898 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:31.861898 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:31.861898 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:31.864891 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:31.865626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.866573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:31.866573 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.866573 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:31.866573 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:31.866573 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.866573 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:31.866573 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.866573 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:31.867552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.867552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:31.867552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:31.867552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:31.867552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.867552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:31.867552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:31.867552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:31.867552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.867552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:31.867552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:31.867552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:31.867552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:31.867552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:31.867552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:31.867873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:31.867873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:31.867873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.867873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:31.867873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:31.867873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.867873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:31.867873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.867873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:31.872129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.872129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:31.872129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:31.872129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:31.872129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.872129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:31.872129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:31.872129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:31.872129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:31.872129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.872129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:31.872613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:31.872613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:31.872613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.872613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:31.872613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:31.872613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:31.872613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:31.872613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:31.872613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:31.872613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:31.872613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:31.872613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.872613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:31.872613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:31.872887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.872887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:31.872887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.872887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:31.872887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.872887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:31.872887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:31.872887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:31.872887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:31.872887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:31.872887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.872887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:31.872887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:31.872887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:31.873128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:31.873553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:31.873553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:31.873553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:31.873553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:31.873553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.873553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:31.873553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:31.873553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.873553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:31.873553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.873553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:31.899304 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:31.899839 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:31.899976 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:31.900174 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:31.901009 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:31.901794 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:31.902340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.903143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:31.903143 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.903143 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:31.903143 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:31.903143 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.903143 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:31.903143 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.903143 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:31.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:31.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:31.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:31.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:31.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:31.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:31.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:31.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:31.904779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.905213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:31.905213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:31.905213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:31.905213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:31.905213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:31.905213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:31.905213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:31.905213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:31.905213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.905213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:31.905213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:31.905213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.905213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:31.905853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.905853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:31.914903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.914903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:31.914903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:31.914903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:31.914903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:31.914903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:31.914903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.914903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:31.914903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:31.914903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.914903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:31.914903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:31.914903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.914903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:31.914903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:31.914903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:31.915323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.915652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:31.915652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.915652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:31.919027 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:31.919346 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:31.919503 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:31.920400 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:31.977412 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:31.978267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.979174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:31.979174 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.979174 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:31.979174 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:31.979174 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.979174 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:31.979174 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.979174 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:31.980130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.060 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.980130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:31.980130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:31.980130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:31.980130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.980130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:31.980130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:31.980130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.980130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:31.980130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:31.980130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:31.980130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.980718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:31.980718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:31.980718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:31.980718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:31.980718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:31.980718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:31.980718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:31.980718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:31.980718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.980718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:31.980718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:31.980718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.980718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:31.981212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.981212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:31.981212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.981212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:31.981212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:31.981212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:31.981212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:31.981212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:31.981212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.981212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:31.981212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:31.981212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.981212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:31.981212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:31.981212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:31.981667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:31.982349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:31.982349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:31.982349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:31.982349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:31.982349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:31.982349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:36.785613 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:36.786270 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:36.788290 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:36.789668 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:36.790297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:36.790887 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:36.791095 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:36.791281 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:36.791448 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:36.791642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:36.791810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:36.792035 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:36.794298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:36.795394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.796583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:36.796583 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.796583 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:36.796583 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:36.796583 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.796583 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:36.796583 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.796583 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:36.797752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.797752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:36.797752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:36.797752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:36.797752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.797752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:36.797752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:36.797752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.797752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:36.797752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:36.797752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:36.797752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.798234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:36.798234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:36.798234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:36.798234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:36.798234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:36.798234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:36.798234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:36.798234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:36.798234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.798234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:36.798234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:36.798234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.798234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:36.798738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.798738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:36.803037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.803037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:36.803037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:36.803037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:36.803037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:36.803037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:36.803037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.803037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:36.803037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:36.803037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.803037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:36.803037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:36.803037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.803037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:36.803037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:36.803037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:36.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.804824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:36.804824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.804824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:36.827041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:36.828020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:36.828020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:36.828020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:32:36.828502 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:36.828872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.832684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:36.833888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:36.833888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:36.833888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:36.833888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:36.833888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:36.833888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:36.833888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:36.833888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.833888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:36.833888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:36.833888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.833888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:36.833888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.833888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:36.833888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.834373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:36.834373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:36.834373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:36.834373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.834373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:36.834373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:36.834373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:32:36.834373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:36.834373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:36.834373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.834373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:36.834373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:36.834373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:36.834803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.834803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:36.834803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:36.834803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:36.834803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:36.834803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:36.834803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:36.834803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:36.834803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:36.834803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.834803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:36.834803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:36.834803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.834803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:36.835203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.835203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:36.835203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.835203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:36.835203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:36.835203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:36.835203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:36.835203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:36.835203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.835203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:36.835203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:36.835203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.835203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:36.835203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:36.835203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:36.835652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:36.836366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:36.836366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:36.836366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.836366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:36.836366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:36.836366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.836366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:36.836366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.836366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:36.861624 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:36.863484 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:36.863484 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:36.863484 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:36.863484 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:36.863484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.867700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:36.867700 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.867700 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:36.867700 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:36.867700 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.867700 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:36.867700 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.867700 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:36.868247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.868247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:36.868247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:36.868247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:36.868247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.868247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:36.868247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:36.868247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:36.868247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.868247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:36.868247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:36.868247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:36.868247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:36.868247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:36.868247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:36.868656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:36.868656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:36.868656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.868656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:36.868656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:36.868656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.868656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:36.868656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.868656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:36.869145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.869145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:36.869145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:36.869145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:36.869145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.869145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:36.869145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:36.869145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:36.869145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:36.869145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.869145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:36.869492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:36.869492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:36.869492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.869492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:36.869492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:36.869492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:36.869492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:36.869492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:36.869492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:36.869492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:36.869492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:36.869492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.869492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:36.869492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:36.869915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.869915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:36.869915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.869915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.871239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:36.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:36.872265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.872265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:36.872265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.872265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:36.899945 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:36.900259 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:36.900329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:36.900608 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:36.900984 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:36.901228 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:36.901558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.902532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:36.902532 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.902532 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:36.902532 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:36.902532 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.902532 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:36.902532 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.902532 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:36.903644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.903644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:36.903644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:36.903644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:36.903644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.903644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:36.903644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:36.903644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.903644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:36.903644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:36.903644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:36.903644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.904154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:36.904154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:36.904154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:36.904154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:36.904154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:36.904154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:36.904154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:36.904154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:36.904154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.904154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:36.904154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:36.904154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.904154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:36.904397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.904397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:36.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:36.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:36.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:36.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:36.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:36.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:36.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:36.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:36.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:36.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:36.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:36.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:36.906786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.908088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:36.908088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.908088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:36.919495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:36.921620 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:36.921620 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:36.921620 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:36.969734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:36.970053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.971644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:36.971644 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.971644 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:36.971644 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:36.971644 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.971644 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:36.971644 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.971644 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:36.972516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.052 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.972516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:36.972516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:36.972516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:36.972516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.972516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:36.972516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:36.972516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.972516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:36.972516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:36.972516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:36.972516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.973028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:36.973028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:36.973028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:36.973028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:36.973028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:36.973028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:36.973028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:36.973028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:36.973028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.973028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:36.973028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:36.973028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.973028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:36.973393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.973393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:36.973630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.973630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:36.973630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:36.973630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:36.973630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:36.973630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:36.973630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.973630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:36.973630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:36.973630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.973630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:36.973630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:36.973630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.973630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:36.973630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:36.973630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:36.974101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:36.974686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:36.974686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:36.974686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:41.786483 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:41.786964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:41.786964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:41.790443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:41.790443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0a4a8800-ed5a-4b20-9ae5-273d694d59a9, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:41.790443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:41.790443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:41.790855 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:41.791128 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:41.792510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:41.792510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:41.792510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:41.792510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7508d7b4-4a45-49ad-82af-9c1d54ec1215, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:41.792510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.793145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:41.793145 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.793145 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:41.793145 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:41.793145 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.793145 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:41.793145 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.793145 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:41.793999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.793999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:41.793999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:41.793999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:41.793999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.793999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:41.793999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:41.793999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.793999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:41.793999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:41.793999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:41.793999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.794481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:41.794481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:41.794481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:41.794481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:41.794481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:41.794481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:41.794481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:41.794481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:41.794481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.794481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:41.794481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:41.794481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.794481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:41.794898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.794898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:41.794898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.794898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:41.794898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:41.794898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:41.794898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:41.794898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:41.794898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.794898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:41.794898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:41.794898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.794898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:41.794898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:41.794898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:41.795378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.795978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:41.795978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:41.795978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.795978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:41.795978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.795978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:41.827898 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:41.828450 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:41.828450 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:41.828630 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:32:41.829864 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:41.829864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.830061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:41.830061 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.830061 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:41.830061 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:41.830061 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.830061 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:41.830061 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.830061 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:41.831218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.831218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:41.831218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:41.831218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:41.831218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.831218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:41.831218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:41.831218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:41.831218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.831218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:41.831218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:41.831218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:41.831218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:41.831218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:41.831218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:41.831606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:41.831606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:41.831606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.831606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:41.831606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:41.831606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.831606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:41.831606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.831606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:41.832071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.832071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:41.832071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:41.832071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:41.832071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.832071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:41.832071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:41.832071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:32:41.832071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:41.832071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:41.832071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.832071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:41.832572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:41.832572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:41.832572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.832572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:41.832572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:41.832572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:41.832572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:41.832572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:41.832572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:41.832572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:41.832572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:41.832572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.832572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:41.832572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:41.833026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.833026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:41.833026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.833026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:41.833274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.833274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:41.833274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:41.833274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:41.833274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:41.833274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:41.833274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.833274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:41.833274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:41.833274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.833274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:41.833274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:41.833274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.833274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:41.833274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:41.833274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:41.833804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:41.834220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.834220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:41.834220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:41.834220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.834220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:41.834220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.834220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:41.862415 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:41.862874 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:41.863061 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:41.863209 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:41.863687 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:41.864472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.864672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:41.864672 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.864672 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:41.864672 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:41.864672 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.864672 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:41.864672 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.864672 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:41.865645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.865645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:41.865645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:41.865645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:41.865645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.865645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:41.865645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:41.865645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:41.865645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.865645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:41.865645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:41.865645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:41.865645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:41.865645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:41.865645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:41.866086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:41.866086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:41.866086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.866086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:41.866086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:41.866086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.866086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:41.866086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.866086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:41.866525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.866525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:41.866525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:41.866525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:41.866525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.866525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:41.866525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:41.866525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:41.866525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:41.866525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.866525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:41.866757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:41.866757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:41.866757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.866757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:41.866757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:41.866757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:41.866757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:41.866757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:41.866757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:41.866757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:41.866757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:41.866757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.866757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:41.866757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:41.867079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.867079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:41.867079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.867079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.870363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:41.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:41.871362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.871362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:41.871362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.871362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:41.900404 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:41.900793 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:41.900939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:41.901133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:41.902181 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:41.902181 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:41.902181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.902412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:41.902412 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.902412 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:41.902412 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:41.902412 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.902412 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:41.902412 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.902412 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:41.905558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.905558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:41.905558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:41.905558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:41.905558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.905558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:41.905558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:41.905558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.905558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:41.905558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:41.905558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:41.905558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.905891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:41.905891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:41.905891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:41.905891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:41.905891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:41.905891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:41.905891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:41.905891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:41.905891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.905891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:41.905891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:41.905891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.905891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:41.906183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.906183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:41.906183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.906183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:41.906183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:41.906183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:41.906183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:41.906183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:41.906183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.906183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:41.906183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:41.906183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.906183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:41.906183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:41.906183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:41.906470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.906871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:41.906871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:41.906871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.906871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:41.906871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.906871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:41.919981 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:41.920340 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:41.920496 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:41.921225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:41.957985 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:41.957985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.958152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:41.958152 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.958152 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:41.958152 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:41.958152 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.958152 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:41.958152 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.958152 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:41.958818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.958818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:41.958818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:41.958818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:41.958818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.958818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:41.958818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:41.958818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.958818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:41.958818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:41.958818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:41.958818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:41.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:41.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:41.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:41.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:41.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:41.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:41.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:41.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:41.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:41.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.959218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:41.959629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.959629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:41.960553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.960553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:41.960553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:41.960553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:41.960553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:41.960553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:41.960553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.960553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:41.960553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:41.960553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.960553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:41.960553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:41.960553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.960553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:41.960553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:41.960553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:41.961064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:41.961685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:41.961685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:41.961685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:46.789164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:46.789892 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:46.789892 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:46.790920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:46.791316 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:46.791577 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:46.792085 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:46.792085 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:46.792211 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:46.792452 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:46.793641 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:46.794068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.795950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:46.795950 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.795950 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:46.795950 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:46.795950 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.795950 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:46.795950 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.795950 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:46.796961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.796961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:46.796961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:46.796961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:46.796961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.796961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:46.796961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:46.796961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.796961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:46.796961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:46.796961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:46.796961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.797488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:46.797488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:46.797488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:46.797488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:46.797488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:46.797488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:46.797488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:46.797488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:46.797488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.797488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:46.797488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:46.797488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.797488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:46.798004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.798004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:46.798097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.798097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:46.798097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:46.798097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:46.798097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:46.798097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:46.798097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.798097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:46.798097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:46.798097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.798097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:46.798097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:46.798097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.798097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:46.798097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:46.798097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:46.799114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.799850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:46.799850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.799850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:46.828362 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:46.830442 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:46.830616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:46.830784 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:32:46.831207 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:46.832818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.833517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:46.833517 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.833517 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:46.833517 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:46.833517 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.833517 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:46.833517 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.833517 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:46.834347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.834347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:46.834347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:46.834347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:46.834347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.834347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:46.834347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:46.834347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:46.834347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.834347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:46.834347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:46.834347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:46.834347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:46.834347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:46.834347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:46.834891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:46.834891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:46.834891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.834891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:46.834891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:46.834891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.834891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:46.834891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.834891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:46.835274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.835274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:46.835274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:46.835274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:46.835274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.835274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:46.835274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:46.835274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:32:46.835274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:46.835274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:46.835274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.835274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:46.835721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:46.835721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:46.835721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.835721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:46.835721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:46.835721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:46.835721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:46.835721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:46.835721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:46.835721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:46.835721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:46.835721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.835721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:46.835721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:46.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:46.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:46.836406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.836406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:46.836406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:46.836406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:46.836406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:46.836406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:46.836406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.836406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:46.836406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:46.836406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.836406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:46.836406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:46.836406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.836406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:46.836406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:46.836406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:46.837633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:46.838235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.838235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:46.838235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:46.838235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.838235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:46.838235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.838235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:46.863482 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:46.863482 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:46.863482 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:46.863482 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:46.864172 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:46.864172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.865156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:46.865156 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.865156 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:46.865156 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:46.865156 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.865156 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:46.865156 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.865156 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:46.865975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.865975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:46.865975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:46.865975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:46.865975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.865975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:46.865975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:46.865975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:46.865975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.865975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:46.865975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:46.865975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:46.865975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:46.865975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:46.865975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:46.866356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:46.866356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:46.866356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.866356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:46.866356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:46.866356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.866356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:46.866356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.866356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:46.866905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.866905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:46.866905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:46.866905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:46.866905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.866905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:46.866905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:46.866905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:46.866905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:46.866905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.866905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:46.867255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:46.867255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:46.867255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.867255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:46.867255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:46.867255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:46.867255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:46.867255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:46.867255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:46.867255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:46.867255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:46.867255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.867255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:46.867255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:46.867617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.867617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:46.867617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.867617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.868035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:46.869293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:46.869904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.869904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:46.869904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.869904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:46.901806 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:46.902116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:46.902258 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:46.902406 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:46.903504 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:46.903775 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:46.904082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.905445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:46.905445 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.905445 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:46.905445 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:46.905445 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.905445 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:46.905445 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.905445 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:46.907619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.907619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:46.907619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:46.907619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:46.907619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.907619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:46.907619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:46.907619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.907619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:46.907619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:46.907619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:46.907619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.908108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:46.908108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:46.908108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:46.908108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:46.908108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:46.908108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:46.908108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:46.908108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:46.908108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.908108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:46.908108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:46.908108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.908108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:46.909221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.909221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:46.909221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.909221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:46.909221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:46.909221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:46.909221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:46.909221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:46.909221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.909221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:46.909221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:46.909221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.909221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:46.909221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:46.909221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:46.909625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.910018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:46.910018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:46.910018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.910018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:46.910018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.910018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:46.920371 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:46.920651 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:46.920749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:46.921357 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:46.957723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:46.957921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.958383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:46.958383 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.958383 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:46.958383 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:46.958383 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.958383 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:46.958383 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.958383 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:46.959126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.959126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:46.959126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:46.959126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:46.959126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.959126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:46.959126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:46.959126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.959126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:46.959126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:46.959126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:46.959126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.959439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:46.959439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:46.959439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:46.959439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:46.959439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:46.959439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:46.959439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:46.959439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:46.959439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.959439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:46.959439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:46.959439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.959439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:46.959771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.959771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:46.959771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.959771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:46.959771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:46.959771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:46.959771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:46.959771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:46.959771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.959771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:46.959771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:46.959771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.959771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:46.959771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:46.959771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:46.960187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:46.960628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:46.960628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:46.960628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:46.960628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:46.960628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:46.960628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:51.789455 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:51.790271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:51.790271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:51.791167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:51.791368 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:51.791524 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:51.791710 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:51.791872 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:51.792082 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:51.792298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:51.792852 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:51.793034 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:51.793203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.793698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:51.793698 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.793698 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:51.793698 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:51.793698 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.793698 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:51.793698 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.793698 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:51.794725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.794725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:51.794725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:51.794725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:51.794725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.794725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:51.794725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:51.794725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.794725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:51.794725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:51.794725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:51.794725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.795196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:51.795196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:51.795196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:51.795196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:51.795196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:51.795196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:51.795196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:51.795196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:51.795196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.795196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:51.795196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:51.795196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.795196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:51.795522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.795522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:51.796009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.796009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:51.796009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:51.796009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:51.796009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:51.796009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:51.796009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.796009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:51.796009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:51.796009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.796009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:51.796009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:51.796009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.796009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:51.796009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:51.796009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:51.796589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.797517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:51.797517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.797517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:51.830550 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:51.831021 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:51.831181 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:51.832287 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:32:51.832741 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:51.833015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.833747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:51.833747 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.833747 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:51.833747 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:51.833747 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.833747 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:51.833747 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.833747 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:51.834918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.834918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:51.834918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:51.834918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:51.834918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.834918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:51.834918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:51.834918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:51.834918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.834918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:51.834918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:51.834918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:51.834918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:51.834918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:51.834918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:51.835576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:51.835576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:51.835576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.835576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:51.835576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:51.835576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.835576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:51.835576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.835576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:51.836815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.836815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:51.836815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:51.836815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:51.836815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.836815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:51.836815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:51.836815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:32:51.836815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:51.836815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:51.836815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.836815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:51.837732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:51.837732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:51.837732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.837732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:51.837732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:51.837732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:51.837732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:51.837732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:51.837732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:51.837732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:51.837732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:51.837732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.837732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:51.837732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:51.838100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.838100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:51.838100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.838100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:51.839446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.839446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:51.839446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:51.839446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:51.839446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:51.839446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:51.839446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.839446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:51.839446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:51.839446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.839446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:51.839446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:51.839446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.839446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:51.839446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:51.839446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:51.840102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:51.840772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.840772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:51.840772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:51.840772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.840772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:51.840772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.840772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:51.863242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:51.864441 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:51.864441 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:51.864441 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:51.864650 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:51.864987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.865577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:51.865577 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.865577 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:51.865577 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:51.865577 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.865577 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:51.865577 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.865577 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:51.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:51.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:51.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:51.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:51.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:51.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:51.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:51.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:51.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:51.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:51.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:51.866643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:51.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:51.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:51.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:51.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:51.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:51.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:51.867500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.867500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:51.867500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:51.867500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:51.867500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.867500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:51.867500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:51.867500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:51.867500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:51.867500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.867500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:51.867830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:51.867830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:51.867830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.867830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:51.867830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:51.867830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:51.867830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:51.867830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:51.867830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:51.867830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:51.867830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:51.867830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.867830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:51.867830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:51.868231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.868231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:51.868231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.868231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.870906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:51.871508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:51.872068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.872068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:51.872068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.872068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:51.902302 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:51.902982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:51.903061 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:51.903278 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:51.903814 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:51.904880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:51.904880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.907349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:51.907349 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.907349 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:51.907349 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:51.907349 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.907349 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:51.907349 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.907349 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:51.908557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.908557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:51.908557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:51.908557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:51.908557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.908557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:51.908557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:51.908557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.908557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:51.908557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:51.908557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:51.908557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.909034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:51.909034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:51.909034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:51.909034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:51.909034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:51.909034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:51.909034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:51.909034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:51.909034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.909034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:51.909034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:51.909034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.909034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:51.909287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.909287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:51.910756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.910756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:51.910756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:51.910756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:51.910756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:51.910756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:51.910756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.910756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:51.910756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:51.910756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.910756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:51.910756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:51.910756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.910756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:51.910756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:51.910756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:51.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.911794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:51.911794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.911794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:51.920906 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:51.921810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:51.922090 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:51.923494 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:51.983796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:51.984146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.984828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:51.984828 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.984828 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:51.984828 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:51.984828 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.984828 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:51.984828 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.984828 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:51.986083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.063 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.986083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:51.986083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:51.986083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:51.986083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.986083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:51.986083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:51.986083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.986083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:51.986083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:51.986083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:51.986083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.987916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:51.987916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:51.987916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:51.987916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:51.987916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:51.987916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:51.987916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:51.987916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:51.987916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.987916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:51.987916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:51.987916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.987916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:51.988321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.988321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:51.988321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.988321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:51.988321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:51.988321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:51.988321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:51.988321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:51.988321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.988321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:51.988321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:51.988321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.988321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:51.988321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:51.988321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:51.988799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:51.989386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:51.989386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:51.989386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:51.989386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:51.989386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:51.989386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:56.790116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:56.794592 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:56.794592 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:56.796705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:56.796705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:56.796705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:56.796705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:56.796990 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:56.797104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:56.797265 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:56.797444 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:56.797585 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:56.797740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.801109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:56.801109 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.801109 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:56.801109 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:56.801109 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.801109 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:56.801109 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.801109 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:56.804166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.804166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:56.804166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:56.804166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:56.804166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.804166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:56.804166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:56.804166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.804166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:56.804166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:56.804166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:56.804166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.804608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:56.804608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:56.804608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:56.804608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:56.804608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:56.804608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:56.804608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:56.804608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:56.804608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.804608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:56.804608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:56.804608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.804608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:56.804949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.804949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:56.806084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.806084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:56.806084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:56.806084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:56.806084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:56.806084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:56.806084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.806084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:56.806084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:56.806084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.806084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:56.806084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:56.806084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.806084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:56.806084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:56.806084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:56.806679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.807142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:56.807142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.807142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:56.833004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:56.833297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:56.833396 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:56.833504 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:32:56.834006 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:56.834199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.834666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:56.834666 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.834666 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:56.834666 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:56.834666 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.834666 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:56.834666 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.834666 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:56.835379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.835379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:56.835379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:56.835379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:56.835379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.835379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:56.835379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:56.835379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:56.835379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.835379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:56.835379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:56.835379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:56.835379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:56.835379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:56.835379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:56.835879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:56.835879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:56.835879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.835879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:56.835879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:56.835879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.835879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:56.835879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.835879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:56.836087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.836087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:56.836087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:56.836087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:56.836087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.836087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:56.836087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:56.836087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:32:56.836087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:56.836087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:56.836087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.836087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:56.836377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:56.836377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:56.836377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.836377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:56.836377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:56.836377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:56.836377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:56.836377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:56.836377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:56.836377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:56.836377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:56.836377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.836377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:56.836377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:56.836752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.836752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:56.836752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.836752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:56.837059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.837059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:56.837059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:56.837059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:56.837059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:56.837059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:56.837059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.837059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:56.837059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:56.837059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.837059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:56.837059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:56.837059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.837059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:56.837059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:56.837059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:56.837488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:56.837908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.837908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:56.837908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:56.837908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.837908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:56.837908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.837908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:56.863776 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:56.864091 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:56.864199 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:56.864348 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:56.864960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:56.865181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.865696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:56.865696 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.865696 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:56.865696 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:56.865696 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.865696 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:56.865696 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.865696 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:56.866683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.866683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:32:56.866683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:56.866683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:56.866683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.866683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:56.866683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:56.866683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:32:56.866683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.866683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:32:56.866683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:56.866683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:32:56.866683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:56.866683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:32:56.866683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:32:56.867170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:56.867170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:32:56.867170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.867170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:32:56.867170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:32:56.867170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.867170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:32:56.867170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.867170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:32:56.868507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.868507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:56.868507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:56.868507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:56.868507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.868507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:56.868507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:56.868507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:56.868507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:56.868507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.868507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:56.869232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:56.869232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:56.869232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.869232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:56.869232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:56.869232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:56.869232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:56.869232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:56.869232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:56.869232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:56.869232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:56.869232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.869232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:56.869232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:56.869682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.869682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:56.869682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.869682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.869898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:56.870422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:56.870789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.870789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:56.870789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.870789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:56.903299 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:56.904745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:56.904969 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:56.905259 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:56.905973 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:56.906283 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:56.906641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.907339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:56.907339 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.907339 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:56.907339 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:56.907339 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.907339 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:56.907339 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.907339 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:56.908457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.908457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:56.908457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:56.908457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:56.908457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.908457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:56.908457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:56.908457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.908457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:56.908457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:56.908457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:56.908457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.908817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:56.908817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:56.908817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:56.908817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:56.908817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:56.908817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:56.908817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:56.908817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:56.908817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.908817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:56.908817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:56.908817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.908817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:56.909133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.909133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:56.909678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.909678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:56.909678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:56.909678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:56.909678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:56.909678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:56.909678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.909678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:56.909678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:56.909678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.909678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:56.909678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:56.909678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.909678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:56.909678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:56.909678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:56.910129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.910573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:56.910573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.910573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:32:56.921504 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:32:56.922078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:32:56.922258 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:32:56.923211 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:32:56.984440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:32:56.987561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.987561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:32:56.987561 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.987561 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:32:56.987561 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:32:56.987561 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.987561 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:32:56.987561 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.987561 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:32:56.987561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.063 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.987561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:32:56.987561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:56.987561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:32:56.987561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.987561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:56.988117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:56.988117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.988117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:56.988117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:56.988117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:32:56.988117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.988117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:32:56.988117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:56.988117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:32:56.988117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:56.988117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:32:56.988117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:32:56.988117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:56.988117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:32:56.988374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.988374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:32:56.988374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:32:56.988374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.988374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:32:56.988374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.988374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:32:56.988374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.988374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:32:56.988374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:32:56.988374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:32:56.988374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:32:56.988374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:32:56.988672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:32:56.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:32:56.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:32:56.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:32:56.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:32:56.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:32:56.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:32:56.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:32:56.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:32:56.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:32:56.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:32:56.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:32:56.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:01.790897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:01.791384 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:01.791384 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:01.792329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:01.792555 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:01.792712 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:01.792822 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:01.792976 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:01.793080 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-499f9be6-39ab-4e07-b298-2e1b161e7a96, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:01.793212 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:01.793361 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:01.793470 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:01.793619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.794258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:01.794258 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.794258 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:01.794258 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:01.794258 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.794258 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:01.794258 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.794258 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:01.795134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.795134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:01.795134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:01.795134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:01.795134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.795134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:01.795134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:01.795134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.795134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:01.795134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:01.795134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:01.795134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.795598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:01.795598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:01.795598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:01.795598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:01.795598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:01.795598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:01.795598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:01.795598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:01.795598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.795598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:01.795598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:01.795598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.795598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:01.796040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.796040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:01.796318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.796318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:01.796318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:01.796318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:01.796318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:01.796318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:01.796318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.796318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:01.796318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:01.796318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.796318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:01.796318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:01.796318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.796318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:01.796318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:01.796318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:01.796688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.797057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:01.797057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.797057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:01.833437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:01.833739 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:01.833852 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:01.834038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:33:01.834559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:01.834757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.835366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:01.835366 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.835366 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:01.835366 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:01.835366 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.835366 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:01.835366 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.835366 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:01.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:01.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:01.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:01.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:01.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:01.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:01.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:01.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:01.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:01.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:01.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:01.836134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:01.836532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:01.836532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:01.836532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.836532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:01.836532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:01.836532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.836532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:01.836532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.836532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:01.836778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.836778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:01.836778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:01.836778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:01.836778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.836778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:01.836778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:01.836778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:33:01.836778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:01.836778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:01.836778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.836778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:01.837112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:01.837112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:01.837112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.837112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:01.837112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:01.837112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:01.837112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:01.837112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:01.837112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:01.837112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:01.837112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:01.837112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.837112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:01.837112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:01.837451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.837451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:01.837451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.837451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:01.837570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.837570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:01.837570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:01.837570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:01.837570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:01.837570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:01.837570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.837570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:01.837570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:01.837570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.837570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:01.837570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:01.837570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.837570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:01.837570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:01.837570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:01.838216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:01.838690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.838690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:01.838690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:01.838690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.838690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:01.838690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.838690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:01.864185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:01.864464 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:01.864621 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:01.864787 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:01.865308 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:01.865521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.866016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:01.866016 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.866016 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:01.866016 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:01.866016 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.866016 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:01.866016 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.866016 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:01.866776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.866776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:01.866776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:01.866776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:01.866776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.866776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:01.866776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:01.866776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:01.866776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.866776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:01.866776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:01.866776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:01.866776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:01.866776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:01.866776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:01.867178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:01.867178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:01.867178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.867178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:01.867178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:01.867178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.867178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:01.867178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.867178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:01.867657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.867657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:01.867657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:01.867657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:01.867657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.867657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:01.867657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:01.867657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:01.867657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:01.867657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.867657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:01.867998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:01.867998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:01.867998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.867998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:01.867998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:01.867998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:01.867998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:01.867998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:01.867998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:01.867998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:01.867998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:01.867998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.867998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:01.867998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:01.868344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.868344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:01.868344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.868344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.868645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:01.869113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:01.869595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.869595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:01.869595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.869595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:01.904595 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:01.904946 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:01.905084 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:01.905296 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:01.905659 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:01.905921 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:01.906148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.906628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:01.906628 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.906628 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:01.906628 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:01.906628 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.906628 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:01.906628 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.906628 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:01.907490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.907490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:01.907490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:01.907490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:01.907490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.907490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:01.907490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:01.907490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.907490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:01.907490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:01.907490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:01.907490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.907854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:01.907854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:01.907854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:01.907854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:01.907854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:01.907854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:01.907854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:01.907854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:01.907854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.907854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:01.907854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:01.907854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.907854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:01.908267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.908267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:01.908350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.908350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:01.908350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:01.908350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:01.908350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:01.908350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:01.908350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.908350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:01.908350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:01.908350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.908350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:01.908350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:01.908350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.908350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:01.908350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:01.908350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:01.908751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.909268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:01.909268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.909268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:01.922026 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:01.922259 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:01.922387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:01.923193 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:01.964450 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:01.964824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.965451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:01.965451 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.965451 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:01.965451 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:01.965451 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.965451 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:01.965451 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.965451 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:01.966419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.966419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:01.966419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:01.966419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:01.966419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.966419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:01.966419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:01.966419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.966419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:01.966419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:01.966419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:01.966419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.966940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:01.966940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:01.966940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:01.966940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:01.966940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:01.966940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:01.966940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:01.966940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:01.966940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.966940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:01.966940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:01.966940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.966940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:01.967384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.967384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:01.967384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.967384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:01.967384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:01.967384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:01.967384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:01.967384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:01.967384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.967384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:01.967384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:01.967384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.967384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:01.967384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:01.967384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:01.967897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:01.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:01.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:01.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:01.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:01.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:01.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:06.792220 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:06.794297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:06.794297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:06.794297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:06.794519 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:06.794582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:06.795922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:06.795922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:06.795922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a5dfdaf3-ab74-439b-908c-33ae5e4d0d1c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:06.795922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:06.795922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:06.795922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.796404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:06.796404 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.796404 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:06.796404 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:06.796404 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.796404 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:06.796404 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.796404 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:06.798404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.798404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:06.798404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:06.798404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:06.798404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.798404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:06.798404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:06.798404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.798404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:06.798404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:06.798404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:06.798404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.798795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:06.798795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:06.798795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:06.798795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:06.798795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:06.798795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:06.798795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:06.798795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:06.798795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.798795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:06.798795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:06.798795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.798795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:06.799100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.799100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:06.800756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.800756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:06.800756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:06.800756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:06.800756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:06.800756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:06.800756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.800756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:06.800756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:06.800756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.800756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:06.800756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:06.800756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.800756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:06.800756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:06.800756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:06.801701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.802122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:06.802122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.802122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:06.833848 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:06.834381 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:06.834590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:06.834761 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:33:06.835352 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:06.835489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.838065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:06.838065 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.838065 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:06.838065 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:06.838065 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.838065 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:06.838065 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.838065 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:06.839206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.839206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:06.839206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:06.839206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:06.839206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.839206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:06.839206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:06.839206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:06.839206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.839206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:06.839206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:06.839206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:06.839206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:06.839206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:06.839206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:06.839726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:06.839726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:06.839726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.839726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:06.839726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:06.839726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.839726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:06.839726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.839726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:06.840329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.840329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:06.840329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:06.840329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:06.840329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.840329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:06.840329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:06.840329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:33:06.840329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:06.840329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:06.840329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.840329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:06.840624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:06.840624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:06.840624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.840624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:06.840624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:06.840624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:06.840624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:06.840624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:06.840624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:06.840624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:06.840624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:06.840624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.840624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:06.840624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:06.841002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.841002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:06.841002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.841002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:06.841604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.841604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:06.841604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:06.841604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:06.841604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:06.841604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:06.841604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.841604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:06.841604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:06.841604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.841604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:06.841604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:06.841604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.841604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:06.841604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:06.841604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:06.843698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:06.844483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.844483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:06.844483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:06.844483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.844483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:06.844483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.844483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:06.864648 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:06.866193 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:06.866315 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:06.866409 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:06.867558 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:06.867897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.868348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:06.868348 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.868348 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:06.868348 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:06.868348 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.868348 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:06.868348 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.868348 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:06.869222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.869222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:06.869222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:06.869222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:06.869222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.869222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:06.869222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:06.869222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:06.869222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.869222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:06.869222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:06.869222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:06.869222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:06.869222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:06.869222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:06.869706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:06.869706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:06.869706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.869706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:06.869706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:06.869706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.869706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:06.869706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.869706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:06.870218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.870218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:06.870218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:06.870218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:06.870218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.870218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:06.870218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:06.870218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:06.870218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:06.870218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.870218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:06.870625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:06.870625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:06.870625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.870625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:06.870625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:06.870625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:06.870625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:06.870625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:06.870625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:06.870625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:06.870625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:06.870625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.870625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:06.870625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:06.871082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.871082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:06.871082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.871082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.871393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:06.871985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:06.873340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.873340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:06.873340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.873340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:06.905106 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:06.905428 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:06.905565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:06.905763 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:06.907382 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:06.907604 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:06.907833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.908323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:06.908323 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.908323 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:06.908323 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:06.908323 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.908323 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:06.908323 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.908323 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:06.909197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.909197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:06.909197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:06.909197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:06.909197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.909197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:06.909197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:06.909197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.909197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:06.909197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:06.909197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:06.909197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.909495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:06.909495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:06.909495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:06.909495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:06.909495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:06.909495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:06.909495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:06.909495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:06.909495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.909495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:06.909495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:06.909495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.909495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:06.909783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.909783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:06.911062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.911062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:06.911062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:06.911062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:06.911062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:06.911062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:06.911062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.911062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:06.911062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:06.911062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.911062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:06.911062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:06.911062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.911062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:06.911062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:06.911062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:06.911438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.911836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:06.911836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.911836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:06.922473 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:06.922722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:06.922806 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:06.923820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:06.969382 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:06.969724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.970519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:06.970519 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.970519 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:06.970519 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:06.970519 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.970519 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:06.970519 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.970519 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:06.971520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.971520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:06.971520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:06.971520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:06.971520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.971520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:06.971520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:06.971520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.971520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:06.971520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:06.971520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:06.971520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.972309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:06.972309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:06.972309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:06.972309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:06.972309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:06.972309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:06.972309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:06.972309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:06.972309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.972309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:06.972309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:06.972309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.972309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:06.972716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.972716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:06.972716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.972716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:06.972716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:06.972716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:06.972716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:06.972716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:06.972716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.972716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:06.972716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:06.972716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.972716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:06.972716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:06.972716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:06.973181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:06.973753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:06.973753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:06.973753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:06.973753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:06.973753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:06.973753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:11.793665 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:11.795463 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:11.795463 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:11.797323 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:11.798713 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:11.800008 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:11.800206 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:11.800380 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:11.800563 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:11.800746 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:11.800825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.803209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:11.803209 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.803209 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:11.803209 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:11.803209 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.803209 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:11.803209 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.803209 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:11.807250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.807250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:11.807250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:11.807250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:11.807250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.807250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:11.807250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:11.807250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.807250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:11.807250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:11.807250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:11.807250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.808150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:11.808150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:11.808150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:11.808150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:11.808150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:11.808150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:11.808150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:11.808150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:11.808150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.808150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:11.808150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:11.808150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.808150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:11.808495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.808495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:11.809991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.809991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:11.809991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:11.809991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:11.809991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:11.809991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:11.809991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.809991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:11.809991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:11.809991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.809991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:11.809991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:11.809991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.809991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:11.809991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:11.809991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:11.810505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.811098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:11.811098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.811098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:11.834521 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:11.834838 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:11.835065 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:11.835303 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:33:11.835588 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:11.836082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.836275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:11.836275 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.836275 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:11.836275 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:11.836275 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.836275 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:11.836275 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.836275 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:11.838245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.838245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:11.838245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:11.838245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:11.838245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.838245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:11.838245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:11.838245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:11.838245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.838245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:11.838245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:11.838245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:11.838245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:11.838245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:11.838245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:11.838833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:11.838833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:11.838833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.838833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:11.838833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:11.838833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.838833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:11.838833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.838833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:11.838833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.838833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:11.838833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:11.838833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:11.838833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.839225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:11.839225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:11.839225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:33:11.839225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:11.839225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:11.839225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.839225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:11.839225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:11.839225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:11.839225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.839225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:11.839225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:11.839225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:11.839617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:11.839617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:11.839617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:11.839617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:11.839617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:11.839617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.839617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:11.839617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:11.839617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.839617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:11.839617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.839617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:11.839847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.839847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:11.839847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:11.839847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:11.839847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:11.839847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:11.839847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.839847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:11.839847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:11.839847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.839847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:11.839847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:11.839847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.839847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:11.839847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:11.839847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:11.840134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:11.840431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.840431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:11.840431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:11.840431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.840431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:11.840431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.840431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:11.866897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:11.866897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:11.866897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:11.867392 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:11.867392 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:11.867726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.868313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:11.868313 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.868313 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:11.868313 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:11.868313 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.868313 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:11.868313 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.868313 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:11.869132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.869132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:11.869132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:11.869132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:11.869132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.869132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:11.869132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:11.869132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:11.869132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.869132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:11.869132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:11.869132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:11.869132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:11.869132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:11.869132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:11.869716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:11.869716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:11.869716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.869716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:11.869716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:11.869716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.869716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:11.869716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.869716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:11.871104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.871104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:11.871104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:11.871104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:11.871104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.871104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:11.871104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:11.871104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:11.871104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:11.871104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.871104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:11.871531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:11.871531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:11.871531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.871531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:11.871531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:11.871531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:11.871531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:11.871531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:11.871531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:11.871531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:11.871531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:11.871531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.871531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:11.871531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:11.871870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.871870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:11.871870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.871870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:11.871870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.871870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:11.871870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:11.871870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:11.871870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:11.871870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:11.871870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.871870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:11.871870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:11.871870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:11.873046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:11.873895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:11.873895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:11.873895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:11.873895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:11.873895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.873895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:11.873895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:11.873895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.873895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:11.873895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.873895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:11.905578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:11.906176 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:11.906312 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:11.906509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:11.906968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:11.907209 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:11.907526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.908306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:11.908306 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.908306 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:11.908306 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:11.908306 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.908306 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:11.908306 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.908306 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:11.909474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.909474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:11.909474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:11.909474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:11.909474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.909474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:11.909474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:11.909474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.909474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:11.909474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:11.909474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:11.909474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.909896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:11.909896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:11.909896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:11.909896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:11.909896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:11.909896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:11.909896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:11.909896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:11.909896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.909896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:11.909896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:11.909896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.909896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:11.910239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.910239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:11.910669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.910669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:11.910669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:11.910669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:11.910669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:11.910669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:11.910669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.910669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:11.910669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:11.910669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.910669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:11.910669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:11.910669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.910669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:11.910669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:11.910669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:11.911148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.911760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:11.911760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.911760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:11.923253 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:11.923331 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:11.923466 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:11.924237 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:11.966767 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:11.967005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.967578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:11.967578 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.967578 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:11.967578 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:11.967578 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.967578 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:11.967578 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.967578 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:11.968331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.968331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:11.968331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:11.968331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:11.968331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.968331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:11.968331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:11.968331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.968331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:11.968331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:11.968331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:11.968331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.968845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:11.968845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:11.968845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:11.968845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:11.968845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:11.968845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:11.968845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:11.968845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:11.968845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.968845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:11.968845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:11.968845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.968845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:11.969288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.969288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:11.969288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.969288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:11.969288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:11.969288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:11.969288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:11.969288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:11.969288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.969288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:11.969288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:11.969288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.969288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:11.969288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:11.969288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:11.969792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:11.970392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:11.970392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:11.970392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:11.970392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:11.970392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:11.970392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:16.795217 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:16.795936 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:16.796041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:16.798205 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:16.798407 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:16.798603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:16.801391 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:16.801523 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:16.801652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:16.801761 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:16.801923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.804597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:16.804597 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:16.804597 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:16.804597 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:16.804597 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:16.804597 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:16.804597 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.804597 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:16.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:16.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:16.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:16.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:16.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:16.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:16.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:16.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:16.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.805693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:16.805693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:16.805693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:16.805693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:16.805693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:16.805693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:16.805693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:16.805693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:16.805693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:16.805693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:16.805693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:16.805693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:16.805693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:16.806046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.806046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:16.806046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.806046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:16.806046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:16.806046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:16.806046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:16.806046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:16.806046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.806046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:16.806046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:16.806046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.806046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:16.806046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:16.806046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:16.806427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:16.806967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:16.806967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:16.806967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:16.806967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:16.806967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.806967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:16.834977 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:16.835262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:16.835370 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:16.835496 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:33:16.836026 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:16.836228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.838493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:16.838493 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:16.838493 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:16.838493 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:16.838493 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:16.838493 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:16.838493 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.838493 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:16.839195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.839195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:16.839195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:16.839195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:16.839195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.839195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:16.839195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:16.839195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:16.839195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.839195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:16.839195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:16.839195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:16.839195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:16.839195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:16.839195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:16.840090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:16.840090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:16.840090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:16.840090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:16.840090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:16.840090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:16.840090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:16.840090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.840090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:16.840090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.840090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:16.840090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:16.840090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:16.840090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.840518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:16.840518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:16.840518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:33:16.840518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:16.840518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:16.840518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.840518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:16.840518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:16.840518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:16.840518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.840518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:16.840518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:16.840518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:16.840969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:16.840969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:16.840969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:16.840969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:16.840969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:16.840969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:16.840969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:16.840969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:16.840969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:16.840969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:16.840969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.840969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:16.841731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.841731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:16.841731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:16.841731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:16.841731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:16.841731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:16.841731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.841731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:16.841731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:16.841731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.841731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:16.841731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:16.841731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.841731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:16.841731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:16.841731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:16.842163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:16.842711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:16.842711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:16.842711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:16.842711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:16.842711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:16.842711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.842711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:16.866770 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:16.867980 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:16.868140 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:16.868291 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:16.869006 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:16.869652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.870255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:16.870255 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:16.870255 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:16.870255 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:16.870255 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:16.870255 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:16.870255 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.870255 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:16.871307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.871307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:16.871307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:16.871307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:16.871307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.871307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:16.871307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:16.871307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:16.871307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.871307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:16.871307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:16.871307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:16.871307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:16.871307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:16.871307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:16.871781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:16.871781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:16.871781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:16.871781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:16.871781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:16.871781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:16.871781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:16.871781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.871781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:16.872353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.872353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:16.872353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:16.872353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:16.872353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.872353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:16.872353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:16.872353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:16.872353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:16.872353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.872353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:16.873611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:16.873611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:16.873611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.873611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:16.873611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:16.873611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:16.873611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:16.873611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:16.873611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:16.873611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:16.873611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:16.873611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:16.873611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:16.873611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:16.874013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:16.874013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:16.874013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.874013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:16.874013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.874013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:16.874013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:16.874013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:16.874013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:16.874013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:16.874013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.874013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:16.874013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:16.874013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:16.874385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:16.875722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:16.875722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:16.875722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:16.875722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:16.875722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:16.875722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:16.875722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:16.875722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:16.875722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:16.875722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.875722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:16.907109 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:16.907109 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:16.907300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:16.907411 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:16.908100 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:16.908271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:16.908550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.910850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:16.910850 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:16.910850 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:16.910850 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:16.910850 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:16.910850 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:16.910850 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.910850 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:16.911871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.911871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:16.911871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:16.911871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:16.911871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.911871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:16.911871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:16.911871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.911871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:16.911871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:16.911871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:16.911871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.912273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:16.912273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:16.912273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:16.912273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:16.912273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:16.912273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:16.912273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:16.912273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:16.912273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:16.912273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:16.912273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:16.912273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:16.912273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:16.912678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.912678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:16.912994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.912994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:16.912994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:16.912994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:16.912994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:16.912994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:16.912994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.912994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:16.912994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:16.912994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.912994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:16.912994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:16.912994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.912994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:16.912994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:16.912994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:16.913563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:16.914390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:16.914390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:16.914390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:16.925226 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:16.926043 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:16.926147 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:16.928014 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:16.999420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:16.999768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:17.000359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:17.000359 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:17.000359 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:17.000359 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:17.000359 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:17.000359 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:17.000359 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:17.000359 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:17.002371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.074 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:17.002371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:17.002371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:17.002371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:17.002371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:17.002371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:17.002371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:17.002371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:17.002371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:17.002371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:17.002371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:17.002371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:17.003183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:17.003183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:17.003183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:17.003183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:17.003183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:17.003183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:17.003183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:17.003183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:17.003183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:17.003183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:17.003183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:17.003183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:17.003183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:17.003592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:17.003592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:17.003664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:17.003664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:17.003664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:17.003664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:17.003664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:17.003664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:17.003664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:17.003664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:17.003664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:17.003664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:17.003664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:17.003664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:17.003664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:17.003664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:17.003664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:17.003664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:17.004329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:17.005681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:17.005681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:17.005681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:21.795767 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:21.796230 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:21.797170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:21.797450 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:21.797681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:21.797879 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:21.797971 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:21.798101 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:21.798237 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:21.798398 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:21.798533 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:21.798707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.799432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:21.799432 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.799432 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:21.799432 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:21.799432 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.799432 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:21.799432 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.799432 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:21.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:21.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:21.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:21.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:21.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:21.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:21.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:21.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:21.800312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.800777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:21.800777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:21.800777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:21.800777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:21.800777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:21.800777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:21.800777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:21.800777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:21.800777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.800777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:21.800777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:21.800777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.800777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:21.801160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.801160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:21.802399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.802399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:21.802399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:21.802399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:21.802399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:21.802399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:21.802399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.802399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:21.802399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:21.802399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.802399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:21.802399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:21.802399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.802399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:21.802399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:21.802399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:21.803011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.803627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:21.803627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.803627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:21.835479 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:21.836262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:21.836443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:21.836673 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:33:21.837115 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:21.837287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.837953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:21.837953 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.837953 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:21.837953 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:21.837953 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.837953 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:21.837953 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.837953 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:21.838704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.838704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:21.838704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:21.838704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:21.838704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.838704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:21.838704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:21.838704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:21.838704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.838704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:21.838704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:21.838704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:21.838704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:21.838704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:21.838704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:21.839220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:21.839220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:21.839220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.839220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:21.839220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:21.839220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.839220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:21.839220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.839220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:21.839656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.839656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:21.839656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:21.839656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:21.839656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.839656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:21.839656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:21.839656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:33:21.839656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:21.839656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:21.839656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.839656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:21.840039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:21.840039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:21.840039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.840039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:21.840039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:21.840039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:21.840039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:21.840039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:21.840039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:21.840039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:21.840039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:21.840039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.840039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:21.840039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:21.840386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.840386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:21.840386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.840386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:21.842654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.842654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:21.842654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:21.842654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:21.842654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:21.842654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:21.842654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.842654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:21.842654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:21.842654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.842654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:21.842654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:21.842654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.842654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:21.842654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:21.842654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:21.843142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:21.843481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.843481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:21.843481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:21.843481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.843481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:21.843481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.843481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:21.868201 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:21.868972 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:21.868972 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:21.869527 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:21.870913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:21.870913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.871624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:21.871624 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.871624 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:21.871624 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:21.871624 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.871624 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:21.871624 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.871624 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:21.876366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.876366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:21.876366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:21.876366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:21.876366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.876366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:21.876366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:21.876366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:21.876366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.876366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:21.876366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:21.876366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:21.876366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:21.876366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:21.876366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:21.876726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:21.876726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:21.876726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.876726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:21.876726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:21.876726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.876726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:21.876726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.876726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:21.876726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.876726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:21.876726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:21.876726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:21.876726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.877090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:21.877090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:21.877090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:21.877090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:21.877090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.877090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:21.877090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:21.877090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:21.877090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.877090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:21.877090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:21.877090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:21.877090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:21.877354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:21.877354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:21.877354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:21.877354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:21.877354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.877354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:21.877354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:21.877354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.877354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:21.877354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.877354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:21.877354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:21.877669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.878173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:21.907349 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:21.908172 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:21.908470 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:21.908779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:21.909275 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:21.909530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:21.909780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.910353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:21.910353 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.910353 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:21.910353 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:21.910353 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.910353 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:21.910353 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.910353 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:21.911472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.911472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:21.911472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:21.911472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:21.911472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.911472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:21.911472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:21.911472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.911472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:21.911472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:21.911472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:21.911472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.911990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:21.911990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:21.911990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:21.911990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:21.911990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:21.911990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:21.911990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:21.911990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:21.911990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.911990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:21.911990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:21.911990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.911990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:21.912400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.912400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:21.914674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.914674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:21.914674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:21.914674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:21.914674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:21.914674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:21.914674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.914674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:21.914674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:21.914674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.914674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:21.914674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:21.914674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.914674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:21.914674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:21.914674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:21.915147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.915467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:21.915467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.915467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:21.926137 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:21.926742 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:21.926865 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:21.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:21.944432 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=241 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:33:21.981086 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:21.981876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.983597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:21.983597 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.983597 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:21.983597 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:21.983597 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.983597 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:21.983597 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.983597 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:21.985971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.985971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:21.985971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:21.985971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:21.985971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.985971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:21.985971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:21.985971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.985971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:21.985971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:21.985971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:21.985971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.986282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:21.986282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:21.986282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:21.986282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:21.986282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:21.986282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:21.986282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:21.986282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:21.986282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.986282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:21.986282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:21.986282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.986282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:21.986567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.986567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:21.986567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.986567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:21.986567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:21.986567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:21.986567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:21.986567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:21.986567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.986567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:21.986567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:21.986567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.986567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:21.986567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:21.986567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:21.986959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:21.987422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:21.987422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:21.987422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:21.987422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:21.987422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:21.987422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:26.797651 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:26.798196 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:26.798276 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:26.800642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:26.800910 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:26.801110 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:26.801468 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:26.801664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:26.801826 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:26.802129 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:26.802323 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:26.802480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.803182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:26.803182 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.803182 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:26.803182 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:26.803182 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.803182 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:26.803182 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.803182 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:26.804113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.804113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:26.804113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:26.804113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:26.804113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.804113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:26.804113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:26.804113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.804113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:26.804113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:26.804113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:26.804113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.804634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:26.804634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:26.804634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:26.804634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:26.804634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:26.804634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:26.804634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:26.804634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:26.804634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.804634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:26.804634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:26.804634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.804634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:26.805157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.805157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:26.805321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.805321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:26.805321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:26.805321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:26.805321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:26.805321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:26.805321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.805321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:26.805321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:26.805321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.805321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:26.805321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:26.805321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.805321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:26.805321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:26.805321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:26.805991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.806709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:26.806709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.806709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:26.836447 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:26.837975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:26.838129 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:26.838277 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:33:26.838861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:26.841655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.842447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:26.842447 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.842447 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:26.842447 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:26.842447 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.842447 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:26.842447 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.842447 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:26.843661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.843661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:26.843661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:26.843661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:26.843661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.843661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:26.843661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:26.843661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:26.843661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.843661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:26.843661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:26.843661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:26.843661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:26.843661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:26.843661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:26.844150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:26.844150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:26.844150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.844150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:26.844150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:26.844150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.844150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:26.844150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.844150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:26.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:26.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:26.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:26.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:26.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:26.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:33:26.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:26.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:26.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.848928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:26.849219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:26.849219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:26.849219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.849219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:26.849219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:26.849219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:26.849219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:26.849219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:26.849219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:26.849219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:26.849219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:26.849219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.849219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:26.849219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:26.849462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.849462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:26.849462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.849462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:26.850011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.850011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:26.850011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:26.850011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:26.850011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:26.850011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:26.850011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.850011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:26.850011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:26.850011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.850011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:26.850011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:26.850011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.850011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:26.850011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:26.850011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:26.850334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:26.850668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.850668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:26.850668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:26.850668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.850668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:26.850668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.850668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:26.868812 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:26.869110 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:26.869225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:26.869333 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:26.869799 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:26.870051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.870505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:26.870505 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.870505 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:26.870505 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:26.870505 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.870505 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:26.870505 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.870505 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:26.871157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.871157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:26.871157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:26.871157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:26.871157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.871157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:26.871157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:26.871157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:26.871157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.871157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:26.871157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:26.871157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:26.871157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:26.871157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:26.871157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:26.871549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:26.871549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:26.871549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.871549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:26.871549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:26.871549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.871549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:26.871549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.871549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:26.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:26.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:26.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:26.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:26.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:26.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:26.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:26.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.871787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:26.872064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:26.872064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:26.872064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.872064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:26.872064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:26.872064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:26.872064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:26.872064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:26.872064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:26.872064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:26.872064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:26.872064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.872064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:26.872064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:26.872476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.872476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:26.872476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.872476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.872647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:26.873381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:26.873870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.873870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:26.873870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.873870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:26.908013 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:26.908447 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:26.908658 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:26.908919 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:26.909587 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:26.909848 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:26.910340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.913231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:26.913231 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.913231 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:26.913231 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:26.913231 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.913231 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:26.913231 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.913231 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:26.913934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.913934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:26.913934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:26.913934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:26.913934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.913934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:26.913934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:26.913934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.913934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:26.913934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:26.913934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:26.913934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.914282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:26.914282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:26.914282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:26.914282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:26.914282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:26.914282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:26.914282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:26.914282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:26.914282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.914282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:26.914282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:26.914282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.914282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:26.914532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.914532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:26.914794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.914794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:26.914794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:26.914794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:26.914794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:26.914794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:26.914794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.914794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:26.914794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:26.914794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.914794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:26.914794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:26.914794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.914794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:26.914794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:26.914794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:26.915183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.915600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:26.915600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.915600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:26.927308 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:26.927740 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:26.927891 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:26.928768 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:26.982249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:26.982704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.983414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:26.983414 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.983414 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:26.983414 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:26.983414 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.983414 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:26.983414 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.983414 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:26.984446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.984446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:26.984446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:26.984446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:26.984446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.984446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:26.984446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:26.984446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.984446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:26.984446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:26.984446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:26.984446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.985118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:26.985118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:26.985118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:26.985118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:26.985118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:26.985118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:26.985118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:26.985118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:26.985118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.985118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:26.985118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:26.985118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.985118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:26.985571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.985571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:26.985571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.985571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:26.985571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:26.985571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:26.985571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:26.985571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:26.985571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.985571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:26.985571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:26.985571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.985571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:26.985571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:26.985571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:26.986178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:26.986884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:26.986884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:26.986884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:26.986884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:26.986884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:26.986884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:27.043776 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:27.044178 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:27.044292 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:27.053047 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 01 11:33:27.053214 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Number of inconsistencies found at create/update: networks=7, security_groups=2, routers=4, ports=1, router_ports=4 {{(pid=89427) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 01 11:33:27.053495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=16 {{(pid=89427) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 01 11:33:27.053495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 0d4134e5-3b3a-49d4-a466-c1b6f993b0ab (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:27.092407 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-0d4134e5-3b3a-49d4-a466-c1b6f993b0ab, resource={'id': '0d4134e5-3b3a-49d4-a466-c1b6f993b0ab', 'name': 'private', 'tenant_id': '43b6be50ae544caeb6a73eced65fb6a7', 'project_id': '43b6be50ae544caeb6a73eced65fb6a7', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['16527f37-b4ae-44e5-9382-abb5e59d1cad', 'a33d59fc-0fd4-45c5-9fff-82704a1922a6'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:23:10Z', 'updated_at': '2026-06-01T11:23:13Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 48058}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.092407 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-0d4134e5-3b3a-49d4-a466-c1b6f993b0ab, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.093200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-bedd3a6a-831e-4f6a-adf7-0842bd968bac, external_ids_update=None, columns={'tag': 48058}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.093200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.093200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:27.093200 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.093200 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:27.093200 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:27.093200 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.093200 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:27.093200 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.093200 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:27.093503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 0d4134e5-3b3a-49d4-a466-c1b6f993b0ab (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.093503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:27.093503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:33:27.093503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:33:27.093503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:33:27.093503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:33:27.093503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:33:27.093503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:27.093503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.093503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.093503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:27.094569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:27.094569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.094569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.094569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:27.094569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:27.094569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:27.094569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:27.094569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:27.094569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:27.094569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:27.094569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.094569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:27.094569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:27.094569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.095027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:27.095027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.095027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:27.095027 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 19da64d5-62e3-4b38-84f1-7d0971806ce9 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:27.122263 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-19da64d5-62e3-4b38-84f1-7d0971806ce9, resource={'id': '19da64d5-62e3-4b38-84f1-7d0971806ce9', 'name': 'shared', 'tenant_id': '43b6be50ae544caeb6a73eced65fb6a7', 'project_id': '43b6be50ae544caeb6a73eced65fb6a7', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['c855657f-8ba2-422e-88fd-c5a71ec03ed7'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:24:23Z', 'updated_at': '2026-06-01T11:24:23Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 19987}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.122263 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-19da64d5-62e3-4b38-84f1-7d0971806ce9, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.122551 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-cb9dd134-19dd-491b-aafb-e4edb81fcf60, external_ids_update=None, columns={'tag': 19987}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.122551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.122551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:27.122551 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.122551 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:27.122551 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:27.122551 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.122551 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:27.122551 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.122551 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:27.123260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 19da64d5-62e3-4b38-84f1-7d0971806ce9 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.123260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:27.123260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:33:27.123260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:33:27.123260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:33:27.123260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:33:27.123260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:33:27.123260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:27.123260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.123260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.123260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:27.123847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:27.123847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.123847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.123847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:27.123847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:27.123847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:27.123847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:27.123847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:27.123847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:27.123847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:27.123847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.123847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:27.123847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:27.123847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.124287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:27.124287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.124287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:27.124287 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 211abfd7-7e80-4864-aebb-afe6cae05b53 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:27.152241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-211abfd7-7e80-4864-aebb-afe6cae05b53, resource={'id': '211abfd7-7e80-4864-aebb-afe6cae05b53', 'name': 'tempest-TestMinimumBasicScenario-1121143663-network', 'tenant_id': '04530a956e424bc9914c6cdf4e9aedce', 'project_id': '04530a956e424bc9914c6cdf4e9aedce', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['ecd063e0-b4ec-40b5-b39f-e2e903e82948'], 'standard_attr_id': 578, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:30:17Z', 'updated_at': '2026-06-01T11:30:18Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 63096}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.152241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-211abfd7-7e80-4864-aebb-afe6cae05b53, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestMinimumBasicScenario-1121143663-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.152557 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-bb952de3-db10-4331-8726-7437c4da39a7, external_ids_update=None, columns={'tag': 63096}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.152557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.152557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:27.152557 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.152557 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:27.152557 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:27.152557 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.152557 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:27.152557 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.152557 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:27.154129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 211abfd7-7e80-4864-aebb-afe6cae05b53 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.154129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:27.154129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:33:27.154129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:33:27.154129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:33:27.154129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:33:27.154129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:33:27.154129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:27.154129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.154129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.154129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:27.154486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:27.154486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.154486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.154486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:27.154486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:27.154486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:27.154486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:27.154486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:27.154486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:27.154486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:27.154486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.154486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:27.154486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:27.154486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.155081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:27.155081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.155081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:27.155081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 252bba3f-354e-4ffe-85cb-c3097263a2dc (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:27.186302 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-252bba3f-354e-4ffe-85cb-c3097263a2dc, resource={'id': '252bba3f-354e-4ffe-85cb-c3097263a2dc', 'name': 'tempest-TestMinimumBasicScenario-762729022-network', 'tenant_id': 'd07f630aec3c4704b186324ee167b7f6', 'project_id': 'd07f630aec3c4704b186324ee167b7f6', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['d5d8fcc2-bca4-4256-930c-64f7c1700fbe'], 'standard_attr_id': 599, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:30:25Z', 'updated_at': '2026-06-01T11:30:27Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 61791}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.187836 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-252bba3f-354e-4ffe-85cb-c3097263a2dc, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestMinimumBasicScenario-762729022-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.187836 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-eb044fae-4f24-4d30-bf1e-baf5fb5c4aaf, external_ids_update=None, columns={'tag': 61791}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.187836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.188128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:27.188128 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.188128 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:27.188128 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:27.188128 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.188128 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:27.188128 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.188128 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:27.188720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 252bba3f-354e-4ffe-85cb-c3097263a2dc (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.188720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:27.188720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:33:27.188720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:33:27.188720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:33:27.188720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:33:27.188720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:33:27.188720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:27.188720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.188720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.188720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:27.189174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:27.189174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.189174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.189174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:27.189174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:27.189174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:27.189174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:27.189174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:27.189174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:27.189174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:27.189174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.189174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:27.189174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:27.189174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.189912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:27.189912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.189912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:27.189912 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 95f61054-db1f-4255-a580-0b88af7426c9 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:27.228002 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-95f61054-db1f-4255-a580-0b88af7426c9, resource={'id': '95f61054-db1f-4255-a580-0b88af7426c9', 'name': 'tempest-VolumesSnapshotTestJSON-727448173-network', 'tenant_id': 'b4d0ae90ef7241a895e5f422f83be570', 'project_id': 'b4d0ae90ef7241a895e5f422f83be570', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['e658a42d-3e87-46f0-93f3-0efef68d955c'], 'standard_attr_id': 527, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:29:48Z', 'updated_at': '2026-06-01T11:29:49Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 56652}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.228002 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-95f61054-db1f-4255-a580-0b88af7426c9, col_values=(('external_ids', {'neutron:network_name': 'tempest-VolumesSnapshotTestJSON-727448173-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.228344 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-837675b1-690c-4907-b6c6-20eb67b5fdb3, external_ids_update=None, columns={'tag': 56652}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.228344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.228344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:27.228344 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.228344 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:27.228344 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:27.228344 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.228344 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:27.228344 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.228344 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:27.229884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 95f61054-db1f-4255-a580-0b88af7426c9 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.229884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:27.229884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:33:27.229884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:33:27.229884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:33:27.229884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:33:27.229884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:33:27.229884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:27.229884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.229884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.229884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:27.230284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:27.230284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.230284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.230284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:27.230284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:27.230284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:27.230284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:27.230284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:27.230284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:27.230284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:27.230284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.230284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:27.230284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:27.230284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.230861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:27.230861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.230861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:27.230861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 992debf6-051a-41c5-a9fa-52b684277a7f (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:27.270289 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-992debf6-051a-41c5-a9fa-52b684277a7f, resource={'id': '992debf6-051a-41c5-a9fa-52b684277a7f', 'name': 'tempest-AttachVolumeShelveTestJSON-1669404623-network', 'tenant_id': '654effff25cd49958476dd3b6eb664d4', 'project_id': '654effff25cd49958476dd3b6eb664d4', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['b23443fd-b684-417e-a4a2-8e67a01c2e88'], 'standard_attr_id': 343, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:27:58Z', 'updated_at': '2026-06-01T11:27:58Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 3885}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.270289 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-992debf6-051a-41c5-a9fa-52b684277a7f, col_values=(('external_ids', {'neutron:network_name': 'tempest-AttachVolumeShelveTestJSON-1669404623-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.270554 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-01c40bf0-bfa1-4441-94ac-f1dabf9bb61a, external_ids_update=None, columns={'tag': 3885}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.270554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.270947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:27.270947 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.270947 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:27.270947 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:27.270947 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.270947 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:27.270947 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.270947 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:27.272557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 992debf6-051a-41c5-a9fa-52b684277a7f (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.272557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:27.272557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:33:27.272557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:33:27.272557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:33:27.272557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:33:27.272557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:33:27.272557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:27.272557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.272557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.272557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:27.272876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:27.272876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.272876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.272876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:27.272876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:27.272876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:27.272876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:27.272876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:27.272876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:27.272876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:27.272876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.272876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:27.272876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:27.272876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.273241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:27.273241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.273241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:27.273575 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource c6c73189-2fc5-412b-9ed5-038efd7341bc (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:27.324400 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, resource={'id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'name': 'public', 'tenant_id': '43b6be50ae544caeb6a73eced65fb6a7', 'project_id': '43b6be50ae544caeb6a73eced65fb6a7', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['21571d58-7621-44df-bdae-bad06600904d', '2bd944db-99d5-4573-8fc8-07e7942a1537'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-01T11:23:15Z', 'updated_at': '2026-06-01T11:23:22Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.324400 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.324671 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-4ecb0f57-2a36-41bc-88d1-5cf4f555b097, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.324671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.324977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:27.324977 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.324977 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:27.324977 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:27.324977 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.324977 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:27.324977 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.324977 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:27.326299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource c6c73189-2fc5-412b-9ed5-038efd7341bc (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.326299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:27.326299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:33:27.326299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:33:27.326299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:33:27.326299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:33:27.326299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:33:27.326299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:27.326299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.326299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.326299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:27.326615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:27.326615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.326615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.326615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:27.326615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:27.326615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:27.326615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:27.326615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:27.326615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:27.326615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:27.326615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.326615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:27.326615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:27.326615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.326920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:27.326920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.326920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:27.326920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 5d6b5c57-c841-4e90-afde-5eab341e3831 (type: security_groups) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:27.386925 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Successfully bumped revision number for resource 5d6b5c57-c841-4e90-afde-5eab341e3831 (type: security_groups) to 3 Jun 01 11:33:27.387106 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource e75cb599-0181-4d23-9fa1-0a4da0b70ab5 (type: security_groups) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:27.421514 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Successfully bumped revision number for resource e75cb599-0181-4d23-9fa1-0a4da0b70ab5 (type: security_groups) to 3 Jun 01 11:33:27.421657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 7cd0498f-79a2-4f6f-a1ef-ea0c25bed969 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:27.529584 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, resource={'id': '7cd0498f-79a2-4f6f-a1ef-ea0c25bed969', 'name': 'tempest-VolumesSnapshotTestJSON-727448173-router', 'project_id': 'b4d0ae90ef7241a895e5f422f83be570', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.98'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::4a'}], 'enable_snat': True}, 'gw_port_id': '551f6501-c620-4497-99e7-f711beeb1a83', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.98', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::4a', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:29:50Z', 'updated_at': '2026-06-01T11:29:53Z', 'revision_number': 3, 'tenant_id': 'b4d0ae90ef7241a895e5f422f83be570'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.530524 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, columns={'external_ids': {'neutron:router_name': 'tempest-VolumesSnapshotTestJSON-727448173-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.531975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "551f6501-c620-4497-99e7-f711beeb1a83"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.532788 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "551f6501-c620-4497-99e7-f711beeb1a83"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.532788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.536119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:27.536119 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.536119 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:27.536119 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:27.536119 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.536119 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:27.536119 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.536119 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:27.536466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router 7cd0498f-79a2-4f6f-a1ef-ea0c25bed969. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.538394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 7cd0498f-79a2-4f6f-a1ef-ea0c25bed969 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.538394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:27.538394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:33:27.538394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:33:27.538394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:33:27.538394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:33:27.538394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:33:27.538394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:33:27.538394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:33:27.538394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:33:27.538394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:33:27.538810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:33:27.538810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:33:27.538810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:27.538810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.538810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.538810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:27.538810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:27.538810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.538810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.538810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:27.538810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:27.538810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:27.538810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:27.538810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:27.539678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:27.539678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:27.539678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.539678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:27.539678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:27.539678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.539678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:27.539678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.539678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:27.539678 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 8e1aee75-6636-4216-bb31-629b4b3c2e46 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:27.647803 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, resource={'id': '8e1aee75-6636-4216-bb31-629b4b3c2e46', 'name': 'tempest-AttachVolumeShelveTestJSON-1669404623-router', 'project_id': '654effff25cd49958476dd3b6eb664d4', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.214'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::a9'}], 'enable_snat': True}, 'gw_port_id': '27175b68-b6e6-4311-a3af-4ef5317e3a54', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.214', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::a9', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:27:59Z', 'updated_at': '2026-06-01T11:28:02Z', 'revision_number': 3, 'tenant_id': '654effff25cd49958476dd3b6eb664d4'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.648382 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, columns={'external_ids': {'neutron:router_name': 'tempest-AttachVolumeShelveTestJSON-1669404623-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.648887 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "27175b68-b6e6-4311-a3af-4ef5317e3a54"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.649254 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "27175b68-b6e6-4311-a3af-4ef5317e3a54"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.649569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.650317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:27.650317 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.650317 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:27.650317 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:27.650317 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.650317 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:27.650317 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.650317 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:27.650624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router 8e1aee75-6636-4216-bb31-629b4b3c2e46. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.652142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 8e1aee75-6636-4216-bb31-629b4b3c2e46 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.652142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:27.652142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:33:27.652142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:33:27.652142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:33:27.652142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:33:27.652142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:33:27.652142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:33:27.652142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:33:27.652142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:33:27.652142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:33:27.652500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:33:27.652500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:33:27.652500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:27.652500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.652500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.652500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:27.652500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:27.652500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.652500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.652500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:27.652500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:27.652500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:27.652500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:27.652500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:27.652887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:27.652887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:27.652887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.652887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:27.652887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:27.652887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.652887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:27.652887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.652887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:27.652887 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 8f8af46b-411c-4d2c-9cdc-9c2c486aef2a (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:27.745790 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, resource={'id': '8f8af46b-411c-4d2c-9cdc-9c2c486aef2a', 'name': 'tempest-TestMinimumBasicScenario-1121143663-router', 'project_id': '04530a956e424bc9914c6cdf4e9aedce', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.71'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::10d'}], 'enable_snat': True}, 'gw_port_id': 'ea6248bd-6d44-441b-a8fc-e1453ecced34', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.71', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::10d', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:30:19Z', 'updated_at': '2026-06-01T11:30:22Z', 'revision_number': 3, 'tenant_id': '04530a956e424bc9914c6cdf4e9aedce'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.746035 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, columns={'external_ids': {'neutron:router_name': 'tempest-TestMinimumBasicScenario-1121143663-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.746035 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "ea6248bd-6d44-441b-a8fc-e1453ecced34"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.746035 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "ea6248bd-6d44-441b-a8fc-e1453ecced34"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.746035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.746035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:27.746035 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.746035 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:27.746035 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:27.746430 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.746430 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:27.746430 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.746430 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:27.746430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router 8f8af46b-411c-4d2c-9cdc-9c2c486aef2a. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.747093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 8f8af46b-411c-4d2c-9cdc-9c2c486aef2a (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.747093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:27.747093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:33:27.747093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:33:27.747093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:33:27.747093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:33:27.747093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:33:27.747093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:33:27.747093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:33:27.747093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:33:27.747093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:33:27.747682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:33:27.747682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:33:27.747682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:27.747682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.747682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.747682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:27.747682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:27.747682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.747682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.747682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:27.747682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:27.747682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:27.747682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:27.747682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:27.748016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:27.748016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:27.748016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.748016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:27.748016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:27.748016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.748016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:27.748016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.748016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:27.748016 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource e88e991e-fb24-4b12-8712-369212ae1041 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:27.851461 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-e88e991e-fb24-4b12-8712-369212ae1041, resource={'id': 'e88e991e-fb24-4b12-8712-369212ae1041', 'name': 'tempest-TestMinimumBasicScenario-762729022-router', 'project_id': 'd07f630aec3c4704b186324ee167b7f6', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.110'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::15'}], 'enable_snat': True}, 'gw_port_id': 'e6f852c7-9dab-4a79-8d53-344636602b2e', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.110', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::15', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:30:28Z', 'updated_at': '2026-06-01T11:30:34Z', 'revision_number': 3, 'tenant_id': 'd07f630aec3c4704b186324ee167b7f6'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.852003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-e88e991e-fb24-4b12-8712-369212ae1041, columns={'external_ids': {'neutron:router_name': 'tempest-TestMinimumBasicScenario-762729022-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.852003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "e6f852c7-9dab-4a79-8d53-344636602b2e"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.852197 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "e6f852c7-9dab-4a79-8d53-344636602b2e"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.852256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.852806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:27.852806 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.852806 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:27.852806 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:27.852806 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.852806 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:27.852806 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.852806 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:27.853031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router e88e991e-fb24-4b12-8712-369212ae1041. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.854227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource e88e991e-fb24-4b12-8712-369212ae1041 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.854227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:27.854227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:33:27.854227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:33:27.854227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:33:27.854227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:33:27.854227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:33:27.854227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:33:27.854227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:33:27.854227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:33:27.854227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:33:27.854623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:33:27.854623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:33:27.854623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:27.854623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.854623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.854623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:27.854623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:27.854623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.854623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.854623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:27.854623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:27.854623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:27.854623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:27.854623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:27.855040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:27.855040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:27.855040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.855040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:27.855040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:27.855040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.855040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:27.855040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.855040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:27.855040 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource c4cb0a9f-d61d-4ab6-be73-ef2cf729d51a (type: ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:27.863632 np0000001841 neutron-ovn-maintenance-worker[89427]: WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Skip fixing resource c4cb0a9f-d61d-4ab6-be73-ef2cf729d51a (type: ports). Resource does not exist in Neutron database anymore: neutron_lib.exceptions.PortNotFound: Port c4cb0a9f-d61d-4ab6-be73-ef2cf729d51a could not be found. Jun 01 11:33:27.863920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 27175b68-b6e6-4311-a3af-4ef5317e3a54 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:27.981638 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-27175b68-b6e6-4311-a3af-4ef5317e3a54, resource={'id': '27175b68-b6e6-4311-a3af-4ef5317e3a54', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:8b:b7:5a', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '8e1aee75-6636-4216-bb31-629b4b3c2e46', 'device_owner': 'network:router_gateway', 'standard_attr_id': 349, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.214'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::a9'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:27:59Z', 'updated_at': '2026-06-01T11:28:03Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.981882 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-27175b68-b6e6-4311-a3af-4ef5317e3a54, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::a9/64', '172.24.5.214/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.982110 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=27175b68-b6e6-4311-a3af-4ef5317e3a54, lrouter_port=lrp-27175b68-b6e6-4311-a3af-4ef5317e3a54, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:27.982277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.982998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:27.982998 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.982998 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:27.982998 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:27.982998 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.982998 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:27.982998 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.982998 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:27.984090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 27175b68-b6e6-4311-a3af-4ef5317e3a54 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.984090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:27.984090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:33:27.984090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:33:27.984090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:33:27.984090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:33:27.984090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:33:27.984090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:27.984090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.984090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.984090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:27.984569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:27.984569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:27.984569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:27.984569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:27.984569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:27.984569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:27.984569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:27.984569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:27.984569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:27.984569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:27.984569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:27.984569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:27.984569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:27.984569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:27.985010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:27.985010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:27.985010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:27.985010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 551f6501-c620-4497-99e7-f711beeb1a83 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:28.080611 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-551f6501-c620-4497-99e7-f711beeb1a83, resource={'id': '551f6501-c620-4497-99e7-f711beeb1a83', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:02:12:4e', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '7cd0498f-79a2-4f6f-a1ef-ea0c25bed969', 'device_owner': 'network:router_gateway', 'standard_attr_id': 540, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.98'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::4a'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:29:50Z', 'updated_at': '2026-06-01T11:29:54Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:28.080887 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-551f6501-c620-4497-99e7-f711beeb1a83, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::4a/64', '172.24.5.98/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:28.081199 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=551f6501-c620-4497-99e7-f711beeb1a83, lrouter_port=lrp-551f6501-c620-4497-99e7-f711beeb1a83, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:28.081424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:28.082032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:28.082032 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:28.082032 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:28.082032 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:28.082032 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:28.082032 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:28.082032 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:28.082032 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:28.082987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 551f6501-c620-4497-99e7-f711beeb1a83 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:28.082987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:28.082987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:33:28.082987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:33:28.082987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:33:28.082987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:33:28.082987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:33:28.082987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:28.082987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:28.082987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:28.082987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:28.083366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:28.083366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:28.083366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:28.083366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:28.083366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:28.083366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:28.083366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:28.083366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:28.083366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:28.083366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:28.083366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:28.083366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:28.083366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:28.083366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:28.083773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:28.083773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:28.083773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:28.083773 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource e6f852c7-9dab-4a79-8d53-344636602b2e (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:28.192402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-e6f852c7-9dab-4a79-8d53-344636602b2e, resource={'id': 'e6f852c7-9dab-4a79-8d53-344636602b2e', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:51:cb:41', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'e88e991e-fb24-4b12-8712-369212ae1041', 'device_owner': 'network:router_gateway', 'standard_attr_id': 607, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.110'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::15'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:30:28Z', 'updated_at': '2026-06-01T11:30:34Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:28.193874 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-e6f852c7-9dab-4a79-8d53-344636602b2e, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-e88e991e-fb24-4b12-8712-369212ae1041'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::15/64', '172.24.5.110/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:28.194245 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=e6f852c7-9dab-4a79-8d53-344636602b2e, lrouter_port=lrp-e6f852c7-9dab-4a79-8d53-344636602b2e, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:28.194445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:28.195109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:28.195109 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:28.195109 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:28.195109 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:28.195109 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:28.195109 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:28.195109 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:28.195109 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:28.196010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource e6f852c7-9dab-4a79-8d53-344636602b2e (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:28.196010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:28.196010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:33:28.196010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:33:28.196010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:33:28.196010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:33:28.196010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:33:28.196010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:28.196010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:28.196010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:28.196010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:28.196295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:28.196295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:28.196295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:28.196295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:28.196295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:28.196295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:28.196295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:28.196295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:28.196295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:28.196295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:28.196295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:28.196295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:28.196295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:28.196295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:28.197096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:28.197096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:28.197096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:28.197096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource ea6248bd-6d44-441b-a8fc-e1453ecced34 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:33:28.288418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-ea6248bd-6d44-441b-a8fc-e1453ecced34, resource={'id': 'ea6248bd-6d44-441b-a8fc-e1453ecced34', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:c0:a6:55', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '8f8af46b-411c-4d2c-9cdc-9c2c486aef2a', 'device_owner': 'network:router_gateway', 'standard_attr_id': 583, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.71'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::10d'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:30:19Z', 'updated_at': '2026-06-01T11:30:23Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:28.288648 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-ea6248bd-6d44-441b-a8fc-e1453ecced34, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.71/24', '2001:db8::10d/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:28.288948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=ea6248bd-6d44-441b-a8fc-e1453ecced34, lrouter_port=lrp-ea6248bd-6d44-441b-a8fc-e1453ecced34, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:28.289136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:28.289582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:28.289582 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:28.289582 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:28.289582 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:28.289582 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:28.289582 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:28.289582 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:28.289582 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:28.290279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource ea6248bd-6d44-441b-a8fc-e1453ecced34 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:28.290279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:28.290279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:33:28.290279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:33:28.290279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:33:28.290279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:33:28.290279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:33:28.290279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:28.290279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:28.290279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:28.290279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:28.290694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:28.290694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:28.290694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:28.290694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:28.290694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:28.290694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:28.290694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:28.290694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:28.290694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:28.290694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:28.290694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:28.290694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:28.290694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:28.290694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:28.291105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:28.291105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:28.291105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:28.291105 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 17a031b8-acae-4000-b4bd-e3c37217c0ad (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:33:28.291105 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(17a031b8-acae-4000-b4bd-e3c37217c0ad) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:33:28.297058 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 18d2afed-700f-49cf-8116-32f3cd754635 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:33:28.297293 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(18d2afed-700f-49cf-8116-32f3cd754635) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:33:28.303526 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 1a8c81ec-dcb6-4d94-b45d-24fe28f180bb (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:33:28.303819 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(1a8c81ec-dcb6-4d94-b45d-24fe28f180bb) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:33:28.308657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 22eee9a1-e45c-4f73-817a-2d5495cff0cd (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:33:28.308891 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(22eee9a1-e45c-4f73-817a-2d5495cff0cd) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:33:28.313396 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 2bd5f178-b9ed-4263-81a2-fe9c17971f46 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:33:28.313654 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(2bd5f178-b9ed-4263-81a2-fe9c17971f46) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:33:28.321985 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 330b735f-303c-4832-8b3b-1d8485a3cbd7 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:33:28.322275 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(330b735f-303c-4832-8b3b-1d8485a3cbd7) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:33:28.327101 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 4768de27-e928-48c1-a50c-fcab084311f8 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:33:28.327361 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(4768de27-e928-48c1-a50c-fcab084311f8) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:33:28.333174 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 86423414-9b62-49f4-865b-039bd23e59a3 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:33:28.333525 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(86423414-9b62-49f4-865b-039bd23e59a3) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:33:28.339278 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 8bc74cd7-ca81-4742-be38-30e0a196b371 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:33:28.339647 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(8bc74cd7-ca81-4742-be38-30e0a196b371) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:33:28.345572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 9f098f8e-6b6f-427c-b314-a5c2876ddba1 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:33:28.345934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(9f098f8e-6b6f-427c-b314-a5c2876ddba1) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:33:28.351844 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource b48453ff-52f7-48f7-abb6-b3592f972ab2 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:33:28.352149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(b48453ff-52f7-48f7-abb6-b3592f972ab2) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:33:28.357234 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource bd1afb9c-ebe4-40ea-8ae2-d4c993bdf085 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:33:28.357521 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(bd1afb9c-ebe4-40ea-8ae2-d4c993bdf085) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:33:28.362218 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource d0ed35c8-82fb-4f4c-9991-d2e2c7d905cf (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:33:28.362463 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(d0ed35c8-82fb-4f4c-9991-d2e2c7d905cf) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:33:28.368487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource da3fef6e-ba6a-4091-9e14-a82952dd8f20 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:33:28.368808 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(da3fef6e-ba6a-4091-9e14-a82952dd8f20) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:33:28.374418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource e4f97c4f-c3c2-4128-b88a-c5e114338f18 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:33:28.374648 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(e4f97c4f-c3c2-4128-b88a-c5e114338f18) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:33:28.379633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource f69c5732-2f48-41f8-b81a-9bb52dbdd456 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:33:28.379869 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(f69c5732-2f48-41f8-b81a-9bb52dbdd456) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:33:28.384423 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_for_inconsistencies finished in 1.340 seconds Jun 01 11:33:31.798228 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:31.798864 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:31.798864 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:31.801590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:31.801590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:31.801590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:31.801590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:31.801590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:31.801827 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:31.801827 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8e1aee75-6636-4216-bb31-629b4b3c2e46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:31.801827 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:31.801827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.801827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:31.801827 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.801827 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:31.801827 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:31.801827 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.801827 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:31.801827 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.801827 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:31.802410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.802410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:31.802410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:31.802410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:31.802410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.802410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:31.802410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:31.802410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.802410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:31.802410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:31.802410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:31.802410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.802814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:31.802814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:31.802814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:31.802814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:31.802814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:31.802814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:31.802814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:31.802814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:31.802814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.802814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:31.802814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:31.802814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.802814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:31.803241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.803241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:31.803241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.803241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:31.803241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:31.803241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:31.803241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:31.803241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:31.803241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.803241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:31.803241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:31.803241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.803241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:31.803241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:31.803241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:31.803666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.804218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:31.804218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:31.804218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.804218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:31.804218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.804218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:31.839074 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:31.839610 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:31.839938 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:31.840205 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:33:31.840818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:31.841925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.842242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:31.842242 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.842242 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:31.842242 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:31.842242 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.842242 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:31.842242 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.842242 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:31.843416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.843416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:31.843416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:31.843416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:31.843416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.843416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:31.843416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:31.843416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:31.843416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.843416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:31.843416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:31.843416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:31.843416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:31.843416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:31.843416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:31.843777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:31.843777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:31.843777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.843777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:31.843777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:31.843777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.843777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:31.843777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.843777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:31.845017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.845017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:31.845017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:31.845017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:31.845017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.845017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:31.845017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:31.845017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:33:31.845017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:31.845017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:31.845017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.845017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:31.845284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:31.845284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:31.845284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.845284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:31.845284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:31.845284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:31.845284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:31.845284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:31.845284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:31.845284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:31.845284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:31.845284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.845284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:31.845284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:31.845525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.845525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:31.845525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.845525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:31.846463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.846463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:31.846463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:31.846463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:31.846463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:31.846463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:31.846463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.846463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:31.846463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:31.846463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.846463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:31.846463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:31.846463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.846463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:31.846463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:31.846463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:31.846990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:31.847374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.847374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:31.847374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:31.847374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.847374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:31.847374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.847374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:31.871305 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:31.871305 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:31.871305 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:31.871305 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:31.872592 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:31.872592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.874155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:31.874155 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.874155 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:31.874155 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:31.874155 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.874155 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:31.874155 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.874155 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:31.877768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.877768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:31.877768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:31.877768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:31.877768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.877768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:31.877768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:31.877768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:31.877768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.877768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:31.877768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:31.877768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:31.877768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:31.877768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:31.877768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:31.878402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:31.878402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:31.878402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.878402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:31.878402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:31.878402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.878402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:31.878402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.878402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:31.879696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.879696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:31.879696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:31.879696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:31.879696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.879696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:31.879696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:31.879696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:31.879696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:31.879696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.879696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:31.880233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:31.880233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:31.880233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.880233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:31.880233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:31.880233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:31.880233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:31.880233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:31.880233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:31.880233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:31.880233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:31.880233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.880233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:31.880233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:31.880654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.880654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:31.880654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.880654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.883763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:31.884452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:31.884984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.884984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:31.884984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.884984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:31.908973 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:31.911128 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:31.911335 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:31.912747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:31.914206 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:31.914695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:31.915167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.915868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:31.915868 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.915868 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:31.915868 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:31.915868 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.915868 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:31.915868 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.915868 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:31.916863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.916863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:31.916863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:31.916863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:31.916863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.916863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:31.916863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:31.916863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.916863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:31.916863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:31.916863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:31.916863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.917117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:31.917117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:31.917117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:31.917117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:31.917117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:31.917117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:31.917117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:31.917117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:31.917117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.917117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:31.917117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:31.917117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.917117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:31.917375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.917375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:31.918469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.918469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:31.918469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:31.918469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:31.918469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:31.918469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:31.918469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.918469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:31.918469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:31.918469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.918469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:31.918469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:31.918469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.918469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:31.918469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:31.918469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:31.918767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.919076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:31.919076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.919076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:31.929332 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:31.929842 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:31.930015 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:31.930790 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:31.978060 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:31.979483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.981309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:31.981309 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.981309 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:31.981309 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:31.981309 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.981309 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:31.981309 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.981309 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:31.984191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.051 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.984191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:31.984191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:31.984191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:31.984191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.984191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:31.984191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:31.984191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.984191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:31.984191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:31.984191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:31.984191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.985042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:31.985042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:31.985042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:31.985042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:31.985042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:31.985042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:31.985042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:31.985042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:31.985042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.985042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:31.985042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:31.985042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.985042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:31.985473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.985473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:31.986335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.986335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:31.986335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:31.986335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:31.986335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:31.986335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:31.986335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.986335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:31.986335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:31.986335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.986335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:31.986335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:31.986335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.986335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:31.986335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:31.986335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:31.986920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:31.987472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:31.987472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:31.987472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:36.798983 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:36.799577 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:36.799577 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:36.801612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:36.801612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:36.801612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:36.801612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:36.801612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:36.801901 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:36.801901 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:36.801901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.802564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:36.802564 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.802564 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:36.802564 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:36.802564 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.802564 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:36.802564 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.802564 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:36.803499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.803499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:36.803499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:36.803499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:36.803499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.803499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:36.803499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:36.803499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.803499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:36.803499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:36.803499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:36.803499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.804085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:36.804085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:36.804085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:36.804085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:36.804085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:36.804085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:36.804085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:36.804085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:36.804085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.804085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:36.804085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:36.804085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.804085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:36.804421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.804421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:36.804421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.804421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:36.804421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:36.804421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:36.804421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:36.804421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:36.804421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.804421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:36.804421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:36.804421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.804421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:36.804421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:36.804421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:36.805320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.805873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:36.805873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:36.805873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.805873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:36.805873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.805873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:36.839812 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:36.840497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:36.840733 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:36.840804 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:33:36.841974 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:36.841974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.842108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:36.842108 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.842108 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:36.842108 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:36.842108 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.842108 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:36.842108 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.842108 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:36.842762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.842762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:36.842762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:36.842762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:36.842762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.842762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:36.842762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:36.842762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:36.842762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.842762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:36.842762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:36.842762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:36.842762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:36.842762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:36.842762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:36.843349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:36.843349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:36.843349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.843349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:36.843349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:36.843349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.843349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:36.843349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.843349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:36.843702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.843702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:36.843702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:36.843702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:36.843702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.843702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:36.843702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:36.843702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:33:36.843702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:36.843702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:36.843702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.843702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:36.844172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:36.844172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:36.844172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.844172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:36.844172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:36.844172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:36.844172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:36.844172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:36.844172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:36.844172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:36.844172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:36.844172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.844172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:36.844172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:36.844615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.844615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:36.844615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.844615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:36.844615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.844615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:36.844615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:36.844615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:36.844615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:36.844615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:36.844615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.844615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:36.844615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:36.844615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:36.845531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:36.846166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:36.846166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:36.846166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:36.846166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:36.846166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:36.846166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.846166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:36.846166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:36.846166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.846166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:36.846166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.846166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:36.873039 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:36.873509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:36.874081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:36.874081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:36.874751 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:36.875590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.875796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:36.875796 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.875796 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:36.875796 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:36.875796 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.875796 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:36.875796 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.875796 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:36.876797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.876797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:36.876797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:36.876797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:36.876797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.876797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:36.876797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:36.876797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:36.876797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.876797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:36.876797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:36.876797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:36.876797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:36.876797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:36.876797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:36.877182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:36.877182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:36.877182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.877182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:36.877182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:36.877182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.877182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:36.877182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.877182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:36.877836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.877836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:36.877836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:36.877836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:36.877836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.877836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:36.877836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:36.877836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:36.877836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:36.877836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.877836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:36.878343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:36.878343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:36.878343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.878343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:36.878343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:36.878343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:36.878343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:36.878343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:36.878343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:36.878343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:36.878343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:36.878343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.878343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:36.878343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:36.878631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.878631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:36.878631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.878631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.880721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:36.881342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:36.881947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.881947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:36.881947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.881947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:36.912899 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:36.913306 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:36.913742 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:36.913742 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:36.914371 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:36.914995 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:36.915412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.916828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:36.916828 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.916828 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:36.916828 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:36.916828 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.916828 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:36.916828 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.916828 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:36.917895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.917895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:36.917895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:36.917895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:36.917895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.917895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:36.917895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:36.917895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.917895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:36.917895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:36.917895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:36.917895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.918312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:36.918312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:36.918312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:36.918312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:36.918312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:36.918312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:36.918312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:36.918312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:36.918312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.918312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:36.918312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:36.918312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.918312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:36.918681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.918681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:36.918984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.918984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:36.918984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:36.918984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:36.918984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:36.918984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:36.918984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.918984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:36.918984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:36.918984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.918984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:36.918984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:36.918984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.918984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:36.918984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:36.918984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:36.919455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.919953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:36.919953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.919953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:36.930202 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:36.931056 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:36.931372 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:36.933557 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:36.987981 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:36.989323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.989608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:36.989608 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.989608 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:36.989608 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:36.989608 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.989608 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:36.989608 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.989608 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:36.990712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.990712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:36.990712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:36.990712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:36.990712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.990712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:36.990712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:36.990712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.990712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:36.990712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:36.990712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:36.990712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.991397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:36.991397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:36.991397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:36.991397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:36.991397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:36.991397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:36.991397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:36.991397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:36.991397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.991397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:36.991397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:36.991397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.991397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:36.992147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.992147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:36.992147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.992147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:36.992147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:36.992147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:36.992147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:36.992147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:36.992147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.992147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:36.992147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:36.992147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.992147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:36.992147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:36.992147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:36.992665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:36.993263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:36.993263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:36.993263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:36.993263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:36.993263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:36.993263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:41.799401 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:41.799962 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:41.800138 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:41.801681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:41.801920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:41.802122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:41.802317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:41.802554 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:41.802726 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:41.802936 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:41.803077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.803693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:41.803693 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.803693 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:41.803693 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:41.803693 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.803693 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:41.803693 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.803693 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:41.804883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.804883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:41.804883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:41.804883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:41.804883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.804883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:41.804883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:41.804883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.804883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:41.804883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:41.804883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:41.804883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.805255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:41.805255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:41.805255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:41.805255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:41.805255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:41.805255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:41.805255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:41.805255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:41.805255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.805255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:41.805255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:41.805255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.805255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:41.805676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.805676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:41.806022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.806022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:41.806022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:41.806022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:41.806022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:41.806022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:41.806022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.806022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:41.806022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:41.806022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.806022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:41.806022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:41.806022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.806022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:41.806022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:41.806022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:41.807051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.808158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:41.808158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.808158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:41.840593 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:41.841163 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:41.842185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:41.842391 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:33:41.842796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:41.843047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.843571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:41.843571 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.843571 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:41.843571 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:41.843571 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.843571 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:41.843571 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.843571 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:41.844223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.844223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:41.844223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:41.844223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:41.844223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.844223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:41.844223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:41.844223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:41.844223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.844223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:41.844223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:41.844223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:41.844223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:41.844223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:41.844223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:41.844636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:41.844636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:41.844636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.844636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:41.844636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:41.844636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.844636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:41.844636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.844636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:41.844879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.844879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:41.844879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:41.844879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:41.844879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.844879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:41.844879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:41.844879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:33:41.844879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:41.844879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:41.844879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.844879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:41.845206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:41.845206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:41.845206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.845206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:41.845206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:41.845206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:41.845206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:41.845206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:41.845206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:41.845206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:41.845206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:41.845206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.845206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:41.845206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:41.845570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.845570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:41.845570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.845570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:41.845570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.845570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:41.845570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:41.845570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:41.845570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:41.845570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:41.845570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.845570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:41.845570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:41.845570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:41.845943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:41.846415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:41.846415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:41.846415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:41.846415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:41.846415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:41.846415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.846415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:41.846415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:41.846415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.846415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:41.846415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.846415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:41.873522 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:41.873984 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:41.874089 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:41.874225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:41.874607 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:41.874894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.875422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:41.875422 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.875422 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:41.875422 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:41.875422 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.875422 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:41.875422 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.875422 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:41.876057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.876057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:41.876057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:41.876057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:41.876057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.876057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:41.876057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:41.876057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:41.876057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.876057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:41.876057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:41.876057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:41.876057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:41.876057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:41.876057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:41.876324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:41.876324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:41.876324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.876324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:41.876324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:41.876324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.876324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:41.876324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.876324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:41.876670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.876670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:41.876670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:41.876670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:41.876670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.876670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:41.876670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:41.876670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:41.876670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:41.876670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.876670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:41.876896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:41.876896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:41.876896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.876896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:41.876896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:41.876896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:41.876896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:41.876896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:41.876896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:41.876896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:41.876896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:41.876896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.876896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:41.876896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:41.877152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.877152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:41.877152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.877152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.877404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:41.877712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:41.878075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.878075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:41.878075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.878075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:41.913789 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:41.913959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:41.913959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:41.914171 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:41.914662 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:41.914931 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:41.915228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.915936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:41.915936 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.915936 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:41.915936 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:41.915936 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.915936 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:41.915936 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.915936 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:41.916523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.916523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:41.916523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:41.916523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:41.916523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.916523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:41.916523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:41.916523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.916523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:41.916523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:41.916523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:41.916523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.917056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:41.917056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:41.917056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:41.917056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:41.917056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:41.917056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:41.917056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:41.917056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:41.917056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.917056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:41.917056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:41.917056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.917056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:41.917291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.917291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:41.917291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.917291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:41.917291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:41.917291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:41.917291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:41.917291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:41.917291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.917291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:41.917291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:41.917291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.917291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:41.917291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:41.917291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:41.917579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.918106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:41.918106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:41.918106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.918106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:41.918106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.918106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:41.930734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:41.931164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:41.931299 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:41.932184 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:41.968551 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:41.969169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.969708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:41.969708 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.969708 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:41.969708 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:41.969708 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.969708 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:41.969708 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.969708 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:41.970617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.970617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:41.970617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:41.970617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:41.970617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.970617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:41.970617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:41.970617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.970617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:41.970617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:41.970617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:41.970617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.971038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:41.971038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:41.971038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:41.971038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:41.971038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:41.971038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:41.971038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:41.971038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:41.971038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.971038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:41.971038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:41.971038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.971038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:41.971355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.971355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:41.971638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.971638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:41.971638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:41.971638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:41.971638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:41.971638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:41.971638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.971638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:41.971638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:41.971638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.971638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:41.971638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:41.971638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.971638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:41.971638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:41.971638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:41.972535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:41.973449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:41.973449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:41.973449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:46.800226 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:46.801250 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:46.801409 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:46.802523 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:46.802861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:46.803138 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:46.803371 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:46.803579 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:46.803747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7cd0498f-79a2-4f6f-a1ef-ea0c25bed969, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:46.803886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:46.804061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.807688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:46.807688 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.807688 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:46.807688 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:46.807688 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.807688 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:46.807688 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.807688 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:46.809663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.809663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:46.809663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:46.809663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:46.809663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.809663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:46.809663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:46.809663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.809663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:46.809663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:46.809663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:46.809663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.810235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:46.810235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:46.810235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:46.810235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:46.810235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:46.810235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:46.810235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:46.810235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:46.810235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.810235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:46.810235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:46.810235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.810235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:46.810472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.810472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:46.811369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.811369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:46.811369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:46.811369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:46.811369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:46.811369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:46.811369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.811369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:46.811369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:46.811369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.811369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:46.811369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:46.811369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.811369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:46.811369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:46.811369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:46.812029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.812495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:46.812495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.812495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:46.841317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:46.842590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:46.842590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:46.842590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:33:46.845680 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:46.846233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.847031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:46.847031 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.847031 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:46.847031 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:46.847031 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.847031 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:46.847031 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.847031 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:46.849560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.849560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:46.849560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:46.849560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:46.849560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.849560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:46.849560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:46.849560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:46.849560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.849560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:46.849560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:46.849560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:46.849560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:46.849560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:46.849560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:46.850067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:46.850067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:46.850067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.850067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:46.850067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:46.850067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.850067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:46.850067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.850067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:46.851332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.851332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:46.851332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:46.851332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:46.851332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.851332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:46.851332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:46.851332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:33:46.851332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:46.851332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:46.851332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.851332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:46.851911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:46.851911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:46.851911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.851911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:46.851911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:46.851911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:46.851911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:46.851911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:46.851911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:46.851911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:46.851911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:46.851911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.851911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:46.851911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:46.852347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.852347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:46.852347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.852347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:46.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:46.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:46.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:46.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:46.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:46.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:46.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:46.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:46.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:46.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:46.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:46.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:46.855754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:46.856455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.856455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:46.856455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:46.856455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.856455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:46.856455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.856455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:46.874114 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:46.874555 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:46.874814 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:46.874998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:46.877138 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:46.877453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.878114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:46.878114 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.878114 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:46.878114 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:46.878114 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.878114 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:46.878114 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.878114 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:46.878892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.878892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:46.878892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:46.878892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:46.878892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.878892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:46.878892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:46.878892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:46.878892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.878892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:46.878892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:46.878892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:46.878892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:46.878892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:46.878892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:46.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:46.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:46.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:46.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:46.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:46.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:46.880279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.880279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:46.880279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:46.880279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:46.880279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.880279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:46.880279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:46.880279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:46.880279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:46.880279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.880279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:46.881052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:46.881052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:46.881052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.881052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:46.881052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:46.881052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:46.881052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:46.881052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:46.881052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:46.881052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:46.881052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:46.881052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.881052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:46.881052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:46.881565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.881565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:46.881565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.881565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.881742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:46.882245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:46.882789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.882789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:46.882789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.882789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:46.915689 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:46.917098 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:46.917861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:46.917861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:46.918239 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:46.918561 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:46.919069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.919645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:46.919645 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.919645 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:46.919645 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:46.919645 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.919645 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:46.919645 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.919645 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:46.920993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.920993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:46.920993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:46.920993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:46.920993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.920993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:46.920993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:46.920993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.920993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:46.920993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:46.920993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:46.920993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.921461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:46.921461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:46.921461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:46.921461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:46.921461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:46.921461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:46.921461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:46.921461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:46.921461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.921461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:46.921461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:46.921461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.921461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:46.921831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.921831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:46.922184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.922184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:46.922184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:46.922184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:46.922184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:46.922184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:46.922184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.922184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:46.922184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:46.922184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.922184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:46.922184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:46.922184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.922184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:46.922184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:46.922184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:46.922765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.923307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:46.923307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.923307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:46.931370 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:46.931743 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:46.931908 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:46.932763 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:46.983745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:46.983889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.984797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:46.984797 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.984797 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:46.984797 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:46.984797 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.984797 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:46.984797 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.984797 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:46.985930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.985930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:46.985930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:46.985930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:46.985930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.985930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:46.985930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:46.985930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.985930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:46.985930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:46.985930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:46.985930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.986340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:46.986340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:46.986340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:46.986340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:46.986340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:46.986340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:46.986340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:46.986340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:46.986340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.986340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:46.986340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:46.986340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.986340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:46.986699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.986699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:46.987147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.987147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:46.987147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:46.987147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:46.987147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:46.987147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:46.987147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.987147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:46.987147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:46.987147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.987147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:46.987147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:46.987147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.987147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:46.987147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:46.987147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:46.987813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:46.988330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:46.988330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:46.988330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:51.801460 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:51.802483 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:51.802483 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:51.803389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:51.804803 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:51.805068 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:51.805345 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:51.805622 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:51.805826 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:51.806074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.810643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:51.810643 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.810643 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:51.810643 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:51.810643 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.810643 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:51.810643 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.810643 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:51.811250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.811250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:51.811250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:51.811250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:51.811250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.811250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:51.811250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:51.811250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.811250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:51.811250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:51.811250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:51.811250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.811866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:51.811866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:51.811866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:51.811866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:51.811866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:51.811866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:51.811866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:51.811866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:51.811866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.811866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:51.811866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:51.811866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.811866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:51.812423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.812423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:51.812423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.812423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:51.812423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:51.812423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:51.812423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:51.812423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:51.812423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.812423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:51.812423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:51.812423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.812423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:51.812423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:51.812423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:51.812964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.813693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:51.813693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:51.813693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.813693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:51.813693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.813693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:51.842858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:51.843366 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:51.843532 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:51.843864 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:33:51.844395 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:51.844741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.845405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:51.845405 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.845405 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:51.845405 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:51.845405 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.845405 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:51.845405 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.845405 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:51.846678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.846678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:51.846678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:51.846678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:51.846678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.846678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:51.846678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:51.846678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:51.846678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.846678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:51.846678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:51.846678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:51.846678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:51.846678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:51.846678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:51.847218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:51.847218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:51.847218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.847218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:51.847218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:51.847218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.847218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:51.847218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.847218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:51.847826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.847826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:51.847826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:51.847826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:51.847826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.847826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:51.847826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:51.847826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:33:51.847826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:51.847826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:51.847826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.847826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:51.848271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:51.848271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:51.848271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.848271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:51.848271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:51.848271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:51.848271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:51.848271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:51.848271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:51.848271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:51.848271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:51.848271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.848271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:51.848271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:51.849026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.849026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:51.849026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.849026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:51.849663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.849663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:51.849663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:51.849663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:51.849663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:51.849663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:51.849663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.849663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:51.849663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:51.849663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.849663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:51.849663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:51.849663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.849663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:51.849663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:51.849663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:51.850086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:51.850495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.850495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:51.850495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:51.850495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.850495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:51.850495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.850495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:51.874742 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:51.875596 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:51.875596 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:51.875806 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:51.876442 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:51.876863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.879634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:51.879634 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.879634 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:51.879634 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:51.879634 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.879634 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:51.879634 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.879634 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:51.880487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.880487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:51.880487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:51.880487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:51.880487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.880487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:51.880487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:51.880487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:51.880487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.880487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:51.880487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:51.880487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:51.880487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:51.880487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:51.880487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:51.880915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:51.880915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:51.880915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.880915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:51.880915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:51.880915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.880915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:51.880915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.880915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:51.881171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.881171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:51.881171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:51.881171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:51.881171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.881171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:51.881171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:51.881171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:51.881171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:51.881171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.881171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:51.881503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:51.881503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:51.881503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.881503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:51.881503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:51.881503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:51.881503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:51.881503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:51.881503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:51.881503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:51.881503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:51.881503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.881503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:51.881503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:51.881788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.881788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:51.881788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.881788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.882891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:51.883601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:51.884640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.884640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:51.884640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.884640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:51.916656 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:51.917103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:51.917178 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:51.917339 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:51.917854 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:51.918078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:51.918343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.918850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:51.918850 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.918850 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:51.918850 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:51.918850 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.918850 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:51.918850 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.918850 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:51.919652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.919652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:51.919652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:51.919652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:51.919652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.919652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:51.919652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:51.919652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.919652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:51.919652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:51.919652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:51.919652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.920064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:51.920064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:51.920064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:51.920064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:51.920064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:51.920064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:51.920064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:51.920064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:51.920064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.920064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:51.920064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:51.920064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.920064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:51.920632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.920632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:51.921016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.921016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:51.921016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:51.921016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:51.921016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:51.921016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:51.921016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.921016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:51.921016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:51.921016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.921016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:51.921016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:51.921016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.921016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:51.921016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:51.921016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:51.921887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.922784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:51.922784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.922784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:51.932016 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:51.932432 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:51.932587 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:51.933380 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:51.994285 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:51.994744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.995374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:51.995374 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.995374 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:51.995374 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:51.995374 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.995374 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:51.995374 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.995374 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:51.996497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.063 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.996497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:51.996497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:51.996497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:51.996497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.996497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:51.996497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:51.996497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.996497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:51.996497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:51.996497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:51.996497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.997042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:51.997042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:51.997042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:51.997042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:51.997042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:51.997042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:51.997042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:51.997042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:51.997042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.997042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:51.997042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:51.997042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.997042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:51.997413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.997413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:51.997558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.997558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:51.997558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:51.997558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:51.997558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:51.997558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:51.997558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.997558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:51.997558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:51.997558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.997558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:51.997558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:51.997558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.997558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:51.997558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:51.997558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:51.998104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:51.998721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:51.998721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:51.998721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:56.802253 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:56.803125 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:56.803125 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:56.803400 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:56.803630 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:56.803757 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:56.803912 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:56.804095 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:56.804157 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:56.804270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.805195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:56.805195 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.805195 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:56.805195 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:56.805195 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.805195 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:56.805195 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.805195 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:56.806044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.806044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:56.806044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:56.806044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:56.806044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.806044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:56.806044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:56.806044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.806044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:56.806044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:56.806044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:56.806044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.806535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:56.806535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:56.806535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:56.806535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:56.806535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:56.806535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:56.806535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:56.806535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:56.806535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.806535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:56.806535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:56.806535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.806535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:56.807293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.807293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:56.807293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.807293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:56.807293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:56.807293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:56.807293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:56.807293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:56.807293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.807293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:56.807293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:56.807293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.807293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:56.807293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:56.807293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:56.808267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.808912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:56.808912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:56.808912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.808912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:56.808912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.808912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:56.843512 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:56.843856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:56.844087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:56.844264 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:33:56.844725 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:56.845113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.845749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:56.845749 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.845749 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:56.845749 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:56.845749 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.845749 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:56.845749 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.845749 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:56.846688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.846688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:56.846688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:56.846688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:56.846688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.846688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:56.846688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:56.846688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:56.846688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.846688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:56.846688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:56.846688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:56.846688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:56.846688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:56.846688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:56.847124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:56.847124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:56.847124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.847124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:56.847124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:56.847124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.847124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:56.847124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.847124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:56.847638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.847638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:56.847638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:56.847638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:56.847638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.847638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:56.847638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:56.847638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:33:56.847638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:56.847638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:56.847638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.847638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:56.848034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:56.848034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:56.848034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.848034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:56.848034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:56.848034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:56.848034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:56.848034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:56.848034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:56.848034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:56.848034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:56.848034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.848034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:56.848034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:56.848310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.848310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:56.848310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.848310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:56.848803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.848803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:56.848803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:56.848803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:56.848803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:56.848803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:56.848803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.848803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:56.848803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:56.848803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.848803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:56.848803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:56.848803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.848803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:56.848803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:56.848803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:56.849421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:56.850022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.850022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:56.850022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:56.850022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.850022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:56.850022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.850022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:56.875526 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:56.878271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:56.878271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:56.878271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:56.879645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:56.879979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.881754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:56.881754 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.881754 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:56.881754 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:56.881754 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.881754 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:56.881754 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.881754 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:56.883073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.883073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:33:56.883073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:56.883073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:56.883073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.883073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:56.883073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:56.883073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:33:56.883073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.883073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:33:56.883073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:56.883073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:33:56.883073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:56.883073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:33:56.883073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:33:56.884824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:56.884824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:33:56.884824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.884824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:33:56.884824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:33:56.884824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.884824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:33:56.884824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.884824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:33:56.884824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.884824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:56.884824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:56.884824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:56.884824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.885402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:56.885402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:56.885402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:56.885402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:56.885402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.885402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:56.885402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:56.885402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:56.885402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.885402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:56.885402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:56.885402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:56.885402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:56.886236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:56.886236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:56.886236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:56.886236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:56.886236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.886236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:56.886236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:56.886236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.886236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:56.886236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.886236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.889062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:56.889771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:56.890370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.890370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:56.890370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.890370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:56.917577 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:56.918214 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:56.918705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:56.918902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:56.919709 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:56.919895 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:56.920176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.920813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:56.920813 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.920813 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:56.920813 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:56.920813 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.920813 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:56.920813 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.920813 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:56.921798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.921798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:56.921798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:56.921798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:56.921798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.921798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:56.921798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:56.921798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.921798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:56.921798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:56.921798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:56.921798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.922234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:56.922234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:56.922234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:56.922234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:56.922234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:56.922234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:56.922234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:56.922234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:56.922234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.922234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:56.922234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:56.922234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.922234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:56.922697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.922697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:56.922919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.922919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:56.922919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:56.922919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:56.922919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:56.922919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:56.922919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.922919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:56.922919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:56.922919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.922919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:56.922919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:56.922919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.922919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:56.922919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:56.922919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:56.923274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.923647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:56.923647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.923647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:33:56.933079 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:33:56.933478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:33:56.933672 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:33:56.934456 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:33:56.974466 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:33:56.974803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.975522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:33:56.975522 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.975522 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:33:56.975522 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:33:56.975522 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.975522 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:33:56.975522 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.975522 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:33:56.976753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.976753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:33:56.976753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:56.976753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:33:56.976753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.976753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:56.976753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:56.976753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.976753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:56.976753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:56.976753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:33:56.976753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.977181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:33:56.977181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:56.977181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:33:56.977181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:56.977181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:33:56.977181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:33:56.977181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:56.977181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:33:56.977181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.977181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:33:56.977181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:33:56.977181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.977181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:33:56.977483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.977483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:33:56.978004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.978004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:33:56.978004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:33:56.978004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:33:56.978004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:33:56.978004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:33:56.978004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.978004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:33:56.978004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:33:56.978004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.978004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:33:56.978004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:33:56.978004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.978004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:33:56.978004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:33:56.978004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:33:56.978783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:33:56.979445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:33:56.979445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:33:56.979445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:01.802512 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:01.803109 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:01.803109 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:01.804896 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:01.804896 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:01.804896 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:01.804896 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:01.804896 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:01.805134 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:01.805176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.806119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:01.806119 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.806119 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:01.806119 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:01.806119 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.806119 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:01.806119 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.806119 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:01.807813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.807813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:01.807813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:01.807813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:01.807813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.807813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:01.807813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:01.807813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.807813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:01.807813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:01.807813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:01.807813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.808269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:01.808269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:01.808269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:01.808269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:01.808269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:01.808269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:01.808269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:01.808269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:01.808269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.808269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:01.808269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:01.808269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.808269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:01.808611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.808611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:01.808611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.808611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:01.808611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:01.808611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:01.808611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:01.808611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:01.808611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.808611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:01.808611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:01.808611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.808611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:01.808611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:01.808611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:01.808880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.809766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:01.809766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:01.809766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.809766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:01.809766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.809766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:01.843905 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:01.844274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:01.844361 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:01.844475 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:34:01.845050 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:01.845276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.845875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:01.845875 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.845875 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:01.845875 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:01.845875 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.845875 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:01.845875 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.845875 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:01.846823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.846823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:01.846823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:01.846823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:01.846823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.846823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:01.846823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:01.846823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:01.846823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.846823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:01.846823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:01.846823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:01.846823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:01.846823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:01.846823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:01.847450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:01.847450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:01.847450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.847450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:01.847450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:01.847450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.847450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:01.847450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.847450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:01.847846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.847846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:01.847846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:01.847846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:01.847846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.847846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:01.847846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:01.847846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:34:01.847846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:01.847846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:01.847846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.847846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:01.848239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:01.848239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:01.848239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.848239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:01.848239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:01.848239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:01.848239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:01.848239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:01.848239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:01.848239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:01.848239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:01.848239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.848239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:01.848239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:01.848717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.848717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:01.848717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.848717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:01.848987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.848987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:01.848987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:01.848987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:01.848987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:01.848987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:01.848987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.848987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:01.848987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:01.848987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.848987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:01.848987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:01.848987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.848987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:01.848987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:01.848987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:01.849533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:01.850273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.850273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:01.850273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:01.850273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.850273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:01.850273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.850273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:01.878245 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:01.878674 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:01.878822 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:01.879002 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:01.881167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:01.881355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.881837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:01.881837 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.881837 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:01.881837 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:01.881837 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.881837 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:01.881837 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.881837 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:01.882718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.882718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:01.882718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:01.882718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:01.882718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.882718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:01.882718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:01.882718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:01.882718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.882718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:01.882718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:01.882718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:01.882718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:01.882718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:01.882718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:01.883168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:01.883168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:01.883168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.883168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:01.883168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:01.883168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.883168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:01.883168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.883168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:01.884073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.884073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:01.884073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:01.884073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:01.884073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.884073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:01.884073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:01.884073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:01.884073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:01.884073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.884073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:01.884430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:01.884430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:01.884430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.884430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:01.884430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:01.884430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:01.884430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:01.884430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:01.884430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:01.884430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:01.884430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:01.884430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.884430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:01.884430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:01.884968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.884968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:01.884968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.884968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.885227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:01.885720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:01.886216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.886216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:01.886216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.886216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:01.918194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:01.919817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:01.919895 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:01.920074 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:01.920389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:01.920617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:01.920808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.923024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:01.923024 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.923024 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:01.923024 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:01.923024 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.923024 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:01.923024 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.923024 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:01.923671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.923671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:01.923671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:01.923671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:01.923671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.923671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:01.923671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:01.923671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.923671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:01.923671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:01.923671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:01.923671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.924025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:01.924025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:01.924025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:01.924025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:01.924025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:01.924025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:01.924025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:01.924025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:01.924025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.924025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:01.924025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:01.924025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.924025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:01.924373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.924373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:01.924373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.924373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:01.924373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:01.924373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:01.924373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:01.924373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:01.924373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.924373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:01.924373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:01.924373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.924373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:01.924373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:01.924373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:01.924816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.925363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:01.925363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:01.925363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.925363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:01.925363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.925363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:01.933566 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:01.933794 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:01.933916 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:01.934526 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:01.979913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:01.980153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.982362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:01.982362 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.982362 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:01.982362 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:01.982362 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.982362 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:01.982362 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.982362 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:01.983751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.983751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:01.983751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:01.983751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:01.983751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.983751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:01.983751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:01.983751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.983751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:01.983751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:01.983751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:01.983751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.984183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:01.984183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:01.984183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:01.984183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:01.984183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:01.984183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:01.984183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:01.984183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:01.984183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.984183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:01.984183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:01.984183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.984183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:01.984653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.984653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:01.984653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.984653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:01.984653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:01.984653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:01.984653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:01.984653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:01.984653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.984653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:01.984653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:01.984653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.984653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:01.984653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:01.984653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:01.985081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:01.985604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:01.985604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:01.985604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:01.985604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:01.985604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:01.985604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:06.804750 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:06.805326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:06.805326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:06.806599 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:06.807342 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:06.808389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:06.808389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:06.808389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:06.808532 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:06.808616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:06.809180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.810679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:06.810679 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.810679 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:06.810679 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:06.810679 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.810679 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:06.810679 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.810679 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:06.812375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.812375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:06.812375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:06.812375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:06.812375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.812375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:06.812375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:06.812375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.812375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:06.812375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:06.812375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:06.812375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.812844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:06.812844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:06.812844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:06.812844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:06.812844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:06.812844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:06.812844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:06.812844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:06.812844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.812844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:06.812844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:06.812844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.812844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:06.813247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.813247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:06.814275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.814275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:06.814275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:06.814275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:06.814275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:06.814275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:06.814275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.814275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:06.814275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:06.814275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.814275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:06.814275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:06.814275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.814275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:06.814275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:06.814275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:06.814737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.815223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:06.815223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.815223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:06.846520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:06.846520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:06.846520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:06.846520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:34:06.846520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:06.846520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.847991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:06.847991 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.847991 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:06.847991 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:06.847991 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.847991 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:06.847991 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.847991 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:06.848848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.848848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:06.848848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:06.848848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:06.848848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.848848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:06.848848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:06.848848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:06.848848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.848848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:06.848848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:06.848848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:06.848848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:06.848848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:06.848848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:06.849519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:06.849519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:06.849519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.849519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:06.849519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:06.849519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.849519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:06.849519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.849519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:06.849865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.849865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:06.849865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:06.849865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:06.849865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.849865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:06.849865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:06.849865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:34:06.849865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:06.849865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:06.849865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.849865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:06.851120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:06.851120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:06.851120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.851120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:06.851120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:06.851120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:06.851120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:06.851120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:06.851120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:06.851120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:06.851120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:06.851120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.851120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:06.851120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:06.851556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.851556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:06.851556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.851556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:06.851814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.851814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:06.851814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:06.851814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:06.851814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:06.851814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:06.851814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.851814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:06.851814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:06.851814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.851814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:06.851814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:06.851814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.851814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:06.851814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:06.851814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:06.852288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:06.852880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.852880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:06.852880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:06.852880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.852880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:06.852880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.852880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:06.878818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:06.879919 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:06.880933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:06.880933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:06.881417 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:06.881823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.885197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:06.885672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:06.885672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:06.885672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:06.885672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:06.885672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:06.885672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:06.885672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:06.885672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.885672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:06.885672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:06.885672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.885672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:06.885672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.885672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:06.885672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.886068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:06.886068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:06.886068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:06.886068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.886068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:06.886068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:06.886068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:06.886068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:06.886068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.886068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:06.886068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:06.886068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:06.886068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.886365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:06.886365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:06.886365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:06.886365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:06.886365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:06.886365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:06.886365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:06.886365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:06.886365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.886365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:06.886365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:06.886365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.886365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:06.886657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.886657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:06.886657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.886657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:06.886657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:06.886657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:06.886657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:06.886657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:06.886657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.886657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:06.886657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:06.886657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.886657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:06.886657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:06.886657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:06.887012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:06.887429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:06.887429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.887429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:06.887429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:06.887429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.887429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:06.887429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.887429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:06.919912 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:06.920371 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:06.920691 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:06.921116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:06.922215 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:06.922440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:06.922659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.925662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:06.925662 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.925662 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:06.925662 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:06.925662 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.925662 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:06.925662 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.925662 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:06.925662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.925662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:06.925662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:06.925662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:06.925662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.925662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:06.925662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:06.926333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.926333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:06.926333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:06.926333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:06.926333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.926333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:06.926333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:06.926333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:06.926333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:06.926333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:06.926333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:06.926333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:06.926333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:06.926333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.926774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:06.926774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:06.926774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.926774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:06.926774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.926774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:06.926774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.926774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:06.926774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:06.926774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:06.926774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:06.926774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:06.926774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.926774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:06.927256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:06.927916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:06.927916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:06.927916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:06.927916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.927916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:06.927916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:06.927916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.927916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:06.927916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.927916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:06.934055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:06.934531 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:06.934887 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:06.935908 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:06.979703 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:06.980072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.980736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:06.980736 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.980736 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:06.980736 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:06.980736 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.980736 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:06.980736 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.980736 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:06.981578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.981578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:06.981578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:06.981578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:06.981578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.981578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:06.981578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:06.981578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.981578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:06.981578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:06.981578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:06.981578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.981942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:06.981942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:06.981942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:06.981942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:06.981942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:06.981942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:06.981942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:06.981942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:06.981942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.981942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:06.981942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:06.981942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.981942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:06.982258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.982258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:06.982373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.982373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:06.982373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:06.982373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:06.982373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:06.982373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:06.982373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.982373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:06.982373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:06.982373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.982373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:06.982373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:06.982373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.982373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:06.982373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:06.982373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:06.982801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:06.983326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:06.983326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:06.983326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:11.805231 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:11.805833 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:11.805961 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:11.807176 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:11.807509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:11.807848 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:11.809649 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:11.809938 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:11.810171 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:11.810372 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:11.810572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:11.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:11.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:11.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:11.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.811329 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:11.813232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.813232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:11.813232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:11.813232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:11.813232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.813232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:11.813232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:11.813232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.813232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:11.813232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:11.813232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:11.813232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.813657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:11.813657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:11.813657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:11.813657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:11.813657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:11.813657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:11.813657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:11.813657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:11.813657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.813657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:11.813657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:11.813657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.813657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:11.814090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.814090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:11.815280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.815280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:11.815280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:11.815280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:11.815280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:11.815280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:11.815280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.815280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:11.815280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:11.815280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.815280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:11.815280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:11.815280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.815280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:11.815280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:11.815280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:11.815768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.816307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:11.816307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.816307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:11.845525 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:11.845824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:11.845969 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:11.846092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:34:11.846701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:11.848216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.849001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:11.849001 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.849001 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:11.849001 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:11.849001 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.849001 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:11.849001 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.849001 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:11.849847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.849847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:11.849847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:11.849847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:11.849847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.849847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:11.849847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:11.849847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:11.849847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.849847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:11.849847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:11.849847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:11.849847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:11.849847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:11.849847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:11.850494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:11.850494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:11.850494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.850494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:11.850494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:11.850494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.850494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:11.850494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.850494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:11.850774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.850774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:11.850774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:11.850774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:11.850774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.850774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:11.850774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:11.850774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:34:11.850774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:11.850774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:11.850774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.850774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:11.851069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:11.851069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:11.851069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.851069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:11.851069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:11.851069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:11.851069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:11.851069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:11.851069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:11.851069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:11.851069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:11.851069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.851069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:11.851069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:11.851366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.851366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:11.851366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.851366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:11.851987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.851987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:11.851987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:11.851987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:11.851987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:11.851987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:11.851987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.851987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:11.851987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:11.851987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.851987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:11.851987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:11.851987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.851987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:11.851987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:11.851987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:11.852459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:11.853044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.853044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:11.853044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:11.853044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.853044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:11.853044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.853044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:11.880968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:11.881283 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:11.881425 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:11.881554 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:11.882148 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:11.882466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.883662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:11.883662 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.883662 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:11.883662 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:11.883662 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.883662 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:11.883662 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.883662 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:11.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:11.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:11.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:11.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:11.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:11.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:11.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:11.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:11.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:11.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:11.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:11.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:11.885106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:11.885106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:11.885106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.885106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:11.885106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:11.885106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.885106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:11.885106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.885106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:11.885832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.885832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:11.885832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:11.885832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:11.885832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.885832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:11.885832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:11.885832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:11.885832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:11.885832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.885832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:11.886255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:11.886255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:11.886255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.886255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:11.886255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:11.886255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:11.886255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:11.886255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:11.886255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:11.886255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:11.886255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:11.886255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.886255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:11.886255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:11.886563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.886563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:11.886563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.886563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.893710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:11.894670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:11.895705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.895705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:11.895705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.895705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:11.920506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:11.922362 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:11.922362 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:11.922362 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:11.924407 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:11.925437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:11.928210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.929353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:11.929353 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.929353 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:11.929353 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:11.929353 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.929353 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:11.929353 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.929353 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:11.930717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.930717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:11.930717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:11.930717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:11.930717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.930717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:11.930717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:11.930717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.930717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:11.930717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:11.930717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:11.930717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.931210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:11.931210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:11.931210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:11.931210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:11.931210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:11.931210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:11.931210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:11.931210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:11.931210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.931210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:11.931210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:11.931210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.931210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:11.931669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.931669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:11.931669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.931669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:11.931669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:11.931669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:11.931669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:11.931669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:11.931669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.931669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:11.931669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:11.931669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.931669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:11.931669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:11.931669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:11.931969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.932928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:11.932928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:11.932928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.932928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:11.932928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.932928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:11.935257 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:11.935495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:11.935621 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:11.936447 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:11.982163 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:11.984478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.987365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:11.987365 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.987365 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:11.987365 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:11.987365 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.987365 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:11.987365 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.987365 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:11.987365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.050 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.987365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:11.987365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:11.987365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:11.987365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.987365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:11.987365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:11.987769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.987769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:11.987769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:11.987769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:11.987769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.987769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:11.987769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:11.987769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:11.987769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:11.987769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:11.987769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:11.987769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:11.987769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:11.987769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.988035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:11.988035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:11.988035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.988035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:11.988035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.988035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:11.988555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.988555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:11.988555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:11.988555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:11.988555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:11.988555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:11.988555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.988555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:11.988555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:11.988555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.988555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:11.988555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:11.988555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.988555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:11.988555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:11.988555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:11.989030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:11.989593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:11.989593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:11.989593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:16.807999 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:16.810062 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:16.810062 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:16.810364 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:16.810654 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:16.810838 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:16.811024 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:16.811184 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:16.811363 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:16.811515 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:16.811704 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:16.811852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.812551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:16.812551 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:16.812551 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:16.812551 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:16.812551 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:16.812551 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:16.812551 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.812551 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:16.816128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.816128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:16.816128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:16.816128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:16.816128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.816128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:16.816128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:16.816128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.816128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:16.816128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:16.816128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:16.816128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.817689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:16.817689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:16.817689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:16.817689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:16.817689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:16.817689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:16.817689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:16.817689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:16.817689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:16.817689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:16.817689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:16.817689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:16.817689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:16.818674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.818674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:16.819632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.819632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:16.819632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:16.819632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:16.819632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:16.819632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:16.819632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.819632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:16.819632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:16.819632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.819632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:16.819632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:16.819632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.819632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:16.819632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:16.819632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:16.820644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:16.821711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:16.821711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.821711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:16.846191 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:16.848209 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:16.848320 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:16.848404 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:34:16.848972 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:16.849239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.849912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:16.849912 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:16.849912 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:16.849912 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:16.849912 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:16.849912 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:16.849912 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.849912 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:16.850813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.850813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:16.850813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:16.850813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:16.850813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.850813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:16.850813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:16.850813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:16.850813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.850813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:16.850813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:16.850813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:16.850813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:16.850813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:16.850813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:16.851588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:16.851588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:16.851588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:16.851588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:16.851588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:16.851588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:16.851588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:16.851588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.851588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:16.854746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.854746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:16.854746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:16.854746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:16.854746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.854746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:16.854746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:16.854746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:34:16.854746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:16.854746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:16.854746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.854746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:16.855762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:16.855762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:16.855762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.855762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:16.855762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:16.855762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:16.855762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:16.855762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:16.855762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:16.855762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:16.855762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:16.855762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:16.855762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:16.855762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:16.856617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:16.856617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:16.856617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.856617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:16.857040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.857040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:16.857040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:16.857040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:16.857040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:16.857040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:16.857040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.857040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:16.857040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:16.857040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.857040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:16.857040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:16.857040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.857040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:16.857040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:16.857040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:16.858130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:16.858791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:16.858791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:16.858791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:16.858791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:16.858791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:16.858791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.858791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:16.882384 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:16.883254 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:16.883386 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:16.883515 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:16.885715 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:16.887390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.891956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:16.891956 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:16.891956 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:16.891956 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:16.891956 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:16.891956 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:16.891956 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.891956 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:16.894021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.894021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:16.894021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:16.894021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:16.894021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.894021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:16.894021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:16.894021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:16.894021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.894021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:16.894021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:16.894021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:16.894021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:16.894021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:16.894021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:16.894474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:16.894474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:16.894474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:16.894474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:16.894474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:16.894474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:16.894474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:16.894474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.894474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:16.895467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.011 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.895467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:16.895467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:16.895467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:16.895467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.895467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:16.895467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:16.895467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:16.895467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:16.895467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.895467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:16.896581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:16.896581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:16.896581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.896581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:16.896581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:16.896581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:16.896581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:16.896581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:16.896581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:16.896581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:16.896581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:16.896581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:16.896581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:16.896581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:16.897243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:16.897243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:16.897243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.897243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.899234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:16.899929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:16.900318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:16.900318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:16.900318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.900318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:16.923746 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:16.925917 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:16.926161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:16.928020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:16.928458 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:16.928854 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:16.929603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.929689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:16.929689 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:16.929689 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:16.929689 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:16.929689 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:16.929689 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:16.929689 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.929689 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:16.930363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.930363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:16.930363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:16.930363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:16.930363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.930363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:16.930363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:16.930363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.930363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:16.930363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:16.930363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:16.930363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.930799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:16.930799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:16.930799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:16.930799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:16.930799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:16.930799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:16.930799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:16.930799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:16.930799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:16.930799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:16.930799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:16.930799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:16.930799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:16.931195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.931195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:16.931195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.931195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:16.931195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:16.931195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:16.931195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:16.931195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:16.931195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.931195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:16.931195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:16.931195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.931195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:16.931195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:16.931195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:16.931612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:16.931988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:16.931988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:16.931988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:16.931988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:16.931988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.931988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:16.935729 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:16.936085 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:16.936834 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:16.936988 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:16.994752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:16.994995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.995802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:16.995802 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:16.995802 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:16.995802 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:16.995802 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:16.995802 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:16.995802 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.995802 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:16.999137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.060 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.999137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:16.999137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:16.999137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:16.999137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:16.999137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:16.999137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:16.999137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.999137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:16.999137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:16.999137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:16.999137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:16.999587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:16.999587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:16.999587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:16.999587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:16.999587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:16.999587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:16.999587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:16.999587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:16.999587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:16.999587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:16.999587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:16.999587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:16.999587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:16.999964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:16.999964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:17.000378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:17.000378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:17.000378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:17.000378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:17.000378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:17.000378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:17.000378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:17.000378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:17.000378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:17.000378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:17.000378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:17.000378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:17.000378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:17.000378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:17.000378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:17.000378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:17.000862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:17.001659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:17.001659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:17.001659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:21.811565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:21.812517 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:21.812838 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:21.814267 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:21.814489 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:21.814656 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:21.814810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:21.814949 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:21.815064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:21.815174 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:21.815275 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:21.815397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.815980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:21.815980 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.815980 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:21.815980 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:21.815980 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.815980 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:21.815980 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.815980 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:21.817391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.817391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:21.817391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:21.817391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:21.817391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.817391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:21.817391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:21.817391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.817391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:21.817391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:21.817391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:21.817391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.817825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:21.817825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:21.817825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:21.817825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:21.817825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:21.817825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:21.817825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:21.817825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:21.817825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.817825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:21.817825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:21.817825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.817825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:21.818341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.818341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:21.821957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.821957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:21.821957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:21.821957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:21.821957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:21.821957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:21.821957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.821957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:21.821957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:21.821957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.821957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:21.821957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:21.821957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.821957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:21.821957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:21.821957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:21.822500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.823080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:21.823080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.823080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:21.848527 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:21.849225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:21.849225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:21.849225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:34:21.849722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:21.850003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.853223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:21.853910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:21.853910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:21.853910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:21.853910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:21.853910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:21.853910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:21.853910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:21.853910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.853910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:21.853910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:21.853910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.853910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:21.853910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.853910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:21.853910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.854315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:21.854315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:21.854315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:21.854315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.854315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:21.854315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:21.854315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:34:21.854315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:21.854315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:21.854315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.854315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:21.854315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:21.854315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:21.854654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.854654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:21.854654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:21.854654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:21.854654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:21.854654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:21.854654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:21.854654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:21.854654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:21.854654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.854654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:21.854654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:21.854654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.854654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:21.855016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.855016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:21.855016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.855016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:21.855016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:21.855016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:21.855016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:21.855016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:21.855016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.855016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:21.855016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:21.855016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.855016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:21.855016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:21.855016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:21.855381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:21.855883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:21.855883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:21.855883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.855883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:21.855883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:21.855883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.855883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:21.855883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.855883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:21.883377 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:21.884249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:21.884249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:21.884249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:21.885376 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:21.885825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.889117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:21.889504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:21.889504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:21.889504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:21.889504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:21.889504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:21.889504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:21.889504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:21.889504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.889504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:21.889504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:21.889504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.889504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:21.889504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.889504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:21.889504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.889810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:21.889810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:21.889810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:21.889810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.889810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:21.889810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:21.889810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:21.889810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:21.889810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.889810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:21.889810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:21.889810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:21.889810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.890042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:21.890042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:21.890042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:21.890042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:21.890042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:21.890042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:21.890042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:21.890042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:21.890042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.890042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:21.890042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:21.890042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.890042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:21.890308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.890308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:21.890308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.890308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:21.890308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:21.890308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:21.890308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:21.890308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:21.890308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.890308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:21.890308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:21.890308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.890308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:21.890308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:21.890308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:21.890659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:21.891075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:21.891075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.891075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:21.891075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:21.891075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.891075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:21.891075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.891075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:21.925842 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:21.926648 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:21.926648 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:21.926648 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:21.926962 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:21.927201 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:21.927472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.929866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:21.929866 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.929866 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:21.929866 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:21.929866 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.929866 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:21.929866 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.929866 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:21.929866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.929866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:21.929866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:21.929866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:21.929866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.929866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:21.929866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:21.930329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.930329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:21.930329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:21.930329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:21.930329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.930329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:21.930329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:21.930329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:21.930329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:21.930329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:21.930329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:21.930329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:21.930329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:21.930329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.930712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:21.930712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:21.930712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.930712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:21.930712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.930712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:21.930712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.930712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:21.930712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:21.930712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:21.930712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:21.930712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:21.930712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.930712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:21.931136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:21.931764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:21.931764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:21.931764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:21.931764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.931764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:21.931764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:21.931764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.931764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:21.931764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.931764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:21.936799 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:21.938958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:21.938958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:21.938958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:21.951383 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=862,neutron:UPDATE=2,neutron:DELETE=16 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:34:21.992633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:21.992837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:21.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:21.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:21.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:21.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:21.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:21.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:21.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:21.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:21.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:21.995036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.995036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:21.995036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:21.995036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:21.995036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.995036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:21.995036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:21.995036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:21.995036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:21.995036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:21.995036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:21.995036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:21.995036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:21.995036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.995346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:21.995346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:21.995346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.995346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:21.995346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.995346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:21.996018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.996018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:21.996018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:21.996018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:21.996018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:21.996018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:21.996018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.996018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:21.996018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:21.996018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.996018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:21.996018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:21.996018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.996018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:21.996018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:21.996018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:21.996303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:21.996668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:21.996668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:21.996668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:26.812508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:26.812946 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:26.813072 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:26.814392 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:26.814647 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:26.814782 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:26.814902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:26.815016 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:26.815144 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:26.815270 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:26.815784 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:26.815784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.816137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:26.816137 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.816137 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:26.816137 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:26.816137 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.816137 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:26.816137 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.816137 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:26.816957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.816957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:26.816957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:26.816957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:26.816957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.816957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:26.816957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:26.816957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.816957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:26.816957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:26.816957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:26.816957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.817192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:26.817192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:26.817192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:26.817192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:26.817192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:26.817192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:26.817192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:26.817192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:26.817192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.817192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:26.817192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:26.817192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.817192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:26.817399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.817399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:26.817729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.817729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:26.817729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:26.817729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:26.817729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:26.817729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:26.817729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.817729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:26.817729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:26.817729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.817729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:26.817729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:26.817729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.817729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:26.817729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:26.817729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:26.818092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.818443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:26.818443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.818443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:26.850791 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:26.851151 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:26.851569 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:26.851569 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:34:26.852191 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:26.852402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.853088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:26.853088 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.853088 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:26.853088 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:26.853088 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.853088 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:26.853088 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.853088 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:26.853923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.853923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:26.853923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:26.853923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:26.853923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.853923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:26.853923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:26.853923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:26.853923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.853923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:26.853923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:26.853923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:26.853923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:26.853923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:26.853923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:26.854238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:26.854238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:26.854238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.854238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:26.854238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:26.854238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.854238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:26.854238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.854238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:26.854661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.854661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:26.854661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:26.854661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:26.854661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.854661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:26.854661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:26.854661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:34:26.854661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:26.854661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:26.854661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.854661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:26.854877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:26.854877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:26.854877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.854877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:26.854877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:26.854877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:26.854877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:26.854877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:26.854877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:26.854877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:26.854877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:26.854877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.854877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:26.854877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:26.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:26.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.855127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:26.855415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.855415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:26.855415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:26.855415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:26.855415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:26.855415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:26.855415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.855415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:26.855415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:26.855415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.855415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:26.855415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:26.855415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.855415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:26.855415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:26.855415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:26.855749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:26.856084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.856084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:26.856084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:26.856084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.856084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:26.856084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.856084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:26.883944 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:26.884194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:26.884312 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:26.884456 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:26.884969 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:26.885182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.885619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:26.885619 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.885619 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:26.885619 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:26.885619 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.885619 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:26.885619 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.885619 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:26.886257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.886257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:26.886257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:26.886257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:26.886257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.886257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:26.886257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:26.886257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:26.886257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.886257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:26.886257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:26.886257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:26.886257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:26.886257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:26.886257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:26.886510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:26.886510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:26.886510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.886510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:26.886510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:26.886510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.886510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:26.886510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.886510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:26.887048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.887048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:26.887048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:26.887048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:26.887048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.887048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:26.887048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:26.887048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:26.887048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:26.887048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.887048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:26.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:26.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:26.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:26.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:26.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:26.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:26.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:26.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:26.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:26.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:26.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:26.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:26.887487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.887487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:26.887487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.887487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.887769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:26.888055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:26.888364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.888364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:26.888364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.888364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:26.926728 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:26.927030 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:26.927131 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:26.927269 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:26.927647 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:26.927952 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:26.928232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.928767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:26.928767 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.928767 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:26.928767 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:26.928767 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.928767 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:26.928767 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.928767 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:26.929378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.929378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:26.929378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:26.929378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:26.929378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.929378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:26.929378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:26.929378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.929378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:26.929378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:26.929378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:26.929378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.929665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:26.929665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:26.929665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:26.929665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:26.929665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:26.929665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:26.929665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:26.929665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:26.929665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.929665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:26.929665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:26.929665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.929665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:26.929902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.929902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:26.930070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.930070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:26.930070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:26.930070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:26.930070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:26.930070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:26.930070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.930070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:26.930070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:26.930070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.930070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:26.930070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:26.930070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.930070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:26.930070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:26.930070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:26.930306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.930622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:26.930622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.930622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:26.937248 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:26.937527 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:26.937671 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:26.938639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:26.978252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:26.978460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.979062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:26.979062 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.979062 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:26.979062 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:26.979062 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.979062 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:26.979062 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.979062 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:26.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:26.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:26.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:26.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:26.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:26.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:26.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:26.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:26.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.980521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:26.980521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:26.980521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:26.980521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:26.980521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:26.980521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:26.980521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:26.980521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:26.980521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.980521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:26.980521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:26.980521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.980521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:26.980781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.980781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:26.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:26.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:26.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:26.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:26.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:26.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:26.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:26.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:26.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:26.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:26.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:26.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:26.981668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:26.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:26.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:26.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:31.813565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:31.814123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:31.814123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:31.816558 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:31.816558 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:31.816558 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:31.816558 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:31.816558 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:31.816685 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:31.816685 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:31.816685 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:31.816685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.816685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:31.816685 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.816685 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:31.816685 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:31.816685 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.816685 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:31.816685 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.816685 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:31.817376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.817376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:31.817376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:31.817376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:31.817376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.817376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:31.817376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:31.817376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.817376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:31.817376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:31.817376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:31.817376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.817643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:31.817643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:31.817643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:31.817643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:31.817643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:31.817643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:31.817643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:31.817643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:31.817643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.817643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:31.817643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:31.817643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.817643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:31.817903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.817903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:31.818639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.818639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:31.818639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:31.818639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:31.818639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:31.818639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:31.818639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.818639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:31.818639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:31.818639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.818639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:31.818639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:31.818639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.818639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:31.818639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:31.818639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:31.818964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.819265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:31.819265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.819265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:31.851232 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:31.851517 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:31.851673 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:31.851824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:34:31.852193 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:31.852395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.853630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:31.853630 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.853630 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:31.853630 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:31.853630 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.853630 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:31.853630 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.853630 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:31.853911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.853911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:31.853911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:31.853911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:31.853911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.853911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:31.853911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:31.853911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:31.853911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.853911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:31.853911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:31.853911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:31.853911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:31.853911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:31.853911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:31.854220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:31.854220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:31.854220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.854220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:31.854220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:31.854220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.854220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:31.854220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.854220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:31.854739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.854739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:31.854739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:31.854739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:31.854739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.854739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:31.854739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:31.854739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:34:31.854739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:31.854739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:31.854739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.854739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:31.855106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:31.855106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:31.855106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.855106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:31.855106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:31.855106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:31.855106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:31.855106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:31.855106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:31.855106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:31.855106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:31.855106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.855106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:31.855106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:31.855345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.855345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:31.855345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.855345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:31.855808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.855808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:31.855808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:31.855808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:31.855808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:31.855808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:31.855808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.855808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:31.855808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:31.855808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.855808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:31.855808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:31.855808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.855808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:31.855808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:31.855808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:31.856362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:31.856974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.856974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:31.856974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:31.856974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.856974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:31.856974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.856974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:31.884359 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:31.884695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:31.884848 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:31.885004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:31.885559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:31.885903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.886485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:31.886485 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.886485 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:31.886485 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:31.886485 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.886485 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:31.886485 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.886485 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:31.887424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.887424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:31.887424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:31.887424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:31.887424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.887424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:31.887424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:31.887424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:31.887424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.887424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:31.887424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:31.887424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:31.887424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:31.887424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:31.887424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:31.887935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:31.887935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:31.887935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.887935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:31.887935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:31.887935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.887935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:31.887935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.887935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:31.888346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.888346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:31.888346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:31.888346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:31.888346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.888346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:31.888346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:31.888346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:31.888346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:31.888346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.888346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:31.888686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:31.888686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:31.888686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.888686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:31.888686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:31.888686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:31.888686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:31.888686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:31.888686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:31.888686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:31.888686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:31.888686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.888686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:31.888686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:31.889072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.889072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:31.889072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.889072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.889418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:31.889870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:31.890314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.890314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:31.890314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.890314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:31.927244 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:31.927700 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:31.927875 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:31.928062 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:31.928553 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:31.928887 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:31.929151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.929694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:31.929694 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.929694 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:31.929694 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:31.929694 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.929694 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:31.929694 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.929694 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:31.930388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.930388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:31.930388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:31.930388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:31.930388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.930388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:31.930388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:31.930388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.930388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:31.930388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:31.930388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:31.930388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.930760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:31.930760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:31.930760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:31.930760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:31.930760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:31.930760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:31.930760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:31.930760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:31.930760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.930760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:31.930760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:31.930760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.930760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:31.931104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.931104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:31.931104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.931104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:31.931104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:31.931104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:31.931104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:31.931104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:31.931104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.931104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:31.931104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:31.931104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.931104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:31.931104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:31.931104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:31.931476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.931992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:31.931992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:31.931992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.931992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:31.931992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.931992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:31.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:31.938178 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:31.938318 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:31.939233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:31.978206 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:31.978471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.979095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:31.979095 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.979095 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:31.979095 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:31.979095 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.979095 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:31.979095 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.979095 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:31.980176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.980176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:31.980176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:31.980176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:31.980176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.980176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:31.980176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:31.980176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.980176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:31.980176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:31.980176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:31.980176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.980499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:31.980499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:31.980499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:31.980499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:31.980499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:31.980499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:31.980499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:31.980499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:31.980499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.980499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:31.980499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:31.980499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.980499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:31.980847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.980847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:31.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:31.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:31.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:31.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:31.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:31.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:31.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:31.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:31.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:31.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:31.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:31.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:31.982403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:31.983046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:31.983046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:31.983046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:36.814366 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:36.816329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:36.816329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:36.819474 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:36.819474 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:36.820258 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:36.820872 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:36.821950 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:36.821950 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:36.822077 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:36.822276 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:36.822638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.823851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:36.823851 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:36.823851 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:36.823851 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:36.823851 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:36.823851 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:36.823851 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.823851 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:36.827182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.827182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:36.827182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:36.827182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:36.827182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.827182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:36.827182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:36.827182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.827182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:36.827182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:36.827182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:36.827182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.828365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:36.828365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:36.828365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:36.828365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:36.828365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:36.828365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:36.828365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:36.828365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:36.828365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:36.828365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:36.828365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:36.828365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:36.828365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:36.831136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.831136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:36.831857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.831857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:36.831857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:36.831857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:36.831857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:36.831857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:36.831857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.831857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:36.831857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:36.831857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.831857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:36.831857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:36.831857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.831857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:36.831857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:36.831857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:36.833762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:36.834187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:36.834187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.834187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:36.851707 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:36.852617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:36.852617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:36.852617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:34:36.854372 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:36.854621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.855650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:36.855650 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:36.855650 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:36.855650 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:36.855650 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:36.855650 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:36.855650 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.855650 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:36.857790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.857790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:36.857790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:36.857790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:36.857790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.857790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:36.857790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:36.857790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:36.857790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.857790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:36.857790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:36.857790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:36.857790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:36.857790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:36.857790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:36.858195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:36.858195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:36.858195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:36.858195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:36.858195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:36.858195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:36.858195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:36.858195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.858195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:36.859282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.859282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:36.859282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:36.859282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:36.859282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.859282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:36.859282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:36.859282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:34:36.859282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:36.859282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:36.859282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.859282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:36.859652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:36.859652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:36.859652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.859652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:36.859652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:36.859652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:36.859652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:36.859652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:36.859652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:36.859652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:36.859652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:36.859652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:36.859652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:36.859652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:36.860052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:36.860052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:36.860052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.860052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:36.860233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.860233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:36.860233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:36.860233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:36.860233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:36.860233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:36.860233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.860233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:36.860233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:36.860233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.860233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:36.860233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:36.860233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.860233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:36.860233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:36.860233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:36.860659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:36.861101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:36.861101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:36.861101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:36.861101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:36.861101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:36.861101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.861101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:36.885825 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:36.886770 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:36.886770 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:36.886770 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:36.886872 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:36.889191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:36.889822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.889822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:36.889822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:36.889822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:36.889822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:36.889822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:36.889822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:36.889822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:36.889822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:36.889822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:36.889822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:36.889822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:36.889822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:36.889822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:36.889822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.889822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:36.890378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.890378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:36.890378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:36.890378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:36.890378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.890378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:36.890378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:36.890378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:36.890378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:36.890378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.890378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:36.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:36.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:36.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:36.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:36.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:36.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:36.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:36.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:36.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:36.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:36.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:36.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:36.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:36.891044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:36.891044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:36.891044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.891044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:36.891044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.891044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:36.891044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:36.891044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:36.891044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:36.891044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:36.891044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.891044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:36.891044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:36.891044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:36.891406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:36.891913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:36.891913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:36.891913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:36.891913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:36.891913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:36.891913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:36.891913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:36.891913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:36.891913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:36.891913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.891913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:36.928717 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:36.928717 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:36.928717 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:36.928717 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:36.929014 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:36.929269 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:36.931564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.931564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:36.931564 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:36.931564 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:36.931564 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:36.931564 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:36.931564 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:36.931564 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.931564 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:36.931564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.931564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:36.931564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:36.931564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:36.931564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.931564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:36.932082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:36.932082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.932082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:36.932082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:36.932082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:36.932082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.932082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:36.932082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:36.932082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:36.932082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:36.932082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:36.932082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:36.932082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:36.932082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:36.932409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:36.932409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:36.932409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:36.932409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:36.932409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:36.932409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.932409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:36.932409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.932409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:36.932409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:36.932409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:36.932409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:36.932409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:36.932741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:36.933229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:36.933229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:36.933229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:36.933229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:36.933229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:36.933229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:36.933229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:36.933229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:36.933229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:36.933229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:36.933229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:36.933229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:36.938233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:36.938813 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:36.938864 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:36.939569 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:37.000362 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:37.000552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:37.001236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:37.001236 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:37.001236 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:37.001236 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:37.001236 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:37.001236 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:37.001236 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:37.001236 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:37.002646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.062 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:37.002646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:37.002646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:37.002646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:37.002646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:37.002646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:37.002646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:37.002646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:37.002646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:37.002646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:37.002646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:37.002646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:37.003068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:37.003068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:37.003068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:37.003068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:37.003068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:37.003068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:37.003068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:37.003068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:37.003068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:37.003068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:37.003068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:37.003068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:37.003068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:37.003437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:37.003437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:37.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:37.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:37.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:37.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:37.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:37.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:37.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:37.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:37.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:37.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:37.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:37.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:37.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:37.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:37.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:37.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:37.005649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:37.006293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:37.006293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:37.006293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:41.815482 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:41.816100 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:41.816100 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:41.817217 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:41.817448 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:41.817658 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:41.817955 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:41.818249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:41.818320 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:41.818433 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:41.818605 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:41.818808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.822822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:41.822822 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.822822 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:41.822822 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:41.822822 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.822822 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:41.822822 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.822822 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:41.823911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.823911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:41.823911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:41.823911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:41.823911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.823911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:41.823911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:41.823911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.823911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:41.823911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:41.823911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:41.823911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.824352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:41.824352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:41.824352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:41.824352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:41.824352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:41.824352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:41.824352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:41.824352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:41.824352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.824352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:41.824352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:41.824352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.824352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:41.824762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.824762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:41.825180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.825180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:41.825180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:41.825180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:41.825180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:41.825180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:41.825180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.825180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:41.825180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:41.825180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.825180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:41.825180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:41.825180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.825180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:41.825180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:41.825180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:41.825935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.826469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:41.826469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.826469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:41.852586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:41.852880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:41.853038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:41.853290 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:34:41.854070 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:41.854241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.857326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:41.857326 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.857326 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:41.857326 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:41.857326 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.857326 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:41.857326 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.857326 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:41.858279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:41.858851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:41.858851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:41.858851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.858851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:41.858851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:41.858851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.858851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:41.858851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.858851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:41.859220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.859220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:41.859220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:41.859220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:41.859220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.859220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:41.859220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:41.859220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:34:41.859220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:41.859220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:41.859220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.859220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:41.859580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:41.859580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:41.859580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.859580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:41.859580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:41.859580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:41.859580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:41.859580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:41.859580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:41.859580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:41.859580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:41.859580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.859580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:41.859580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:41.859965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.859965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:41.859965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.859965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:41.860337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.860337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:41.860337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:41.860337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:41.860337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:41.860337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:41.860337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.860337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:41.860337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:41.860337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.860337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:41.860337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:41.860337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.860337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:41.860337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:41.860337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:41.860877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:41.861379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.861379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:41.861379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:41.861379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.861379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:41.861379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.861379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:41.886333 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:41.886474 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:41.886601 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:41.887090 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:41.887090 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:41.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.887733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:41.887733 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.887733 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:41.887733 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:41.887733 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.887733 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:41.887733 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.887733 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:41.888359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.888359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:41.888359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:41.888359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:41.888359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.888359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:41.888359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:41.888359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:41.888359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.888359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:41.888359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:41.888359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:41.888359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:41.888359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:41.888359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:41.888799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:41.888799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:41.888799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.888799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:41.888799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:41.888799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.888799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:41.888799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.888799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:41.889222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.889222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:41.889222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:41.889222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:41.889222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.889222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:41.889222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:41.889222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:41.889222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:41.889222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.889222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:41.889437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:41.889437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:41.889437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.889437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:41.889437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:41.889437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:41.889437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:41.889437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:41.889437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:41.889437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:41.889437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:41.889437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.889437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:41.889437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:41.889702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.889702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:41.889702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.889702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.890343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:41.890706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:41.891081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.891081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:41.891081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.891081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:41.929110 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:41.929724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:41.929848 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:41.930964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:41.930964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:41.930964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:41.930964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.931140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:41.931140 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.931140 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:41.931140 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:41.931140 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.931140 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:41.931140 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.931140 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:41.932600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.932600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:41.932600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:41.932600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:41.932600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.932600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:41.932600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:41.932600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.932600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:41.932600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:41.932600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:41.932600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.933515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:41.933515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:41.933515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:41.933515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:41.933515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:41.933515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:41.933515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:41.933515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:41.933515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.933515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:41.933515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:41.933515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.933515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:41.935416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.935416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:41.935416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.935416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:41.935416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:41.935416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:41.935416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:41.935416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:41.935416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.935416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:41.935416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:41.935416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.935416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:41.935416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:41.935416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:41.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.936483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:41.936483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:41.936483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.936483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:41.936483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.936483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:41.939572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:41.939572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:41.939572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:41.939572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:41.980877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:41.981063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.981631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:41.981631 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.981631 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:41.981631 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:41.981631 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.981631 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:41.981631 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.981631 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:41.982477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.982477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:41.982477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:41.982477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:41.982477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.982477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:41.982477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:41.982477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.982477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:41.982477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:41.982477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:41.982477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.982873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:41.982873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:41.982873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:41.982873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:41.982873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:41.982873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:41.982873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:41.982873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:41.982873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.982873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:41.982873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:41.982873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.982873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:41.983209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.983209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:41.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:41.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:41.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:41.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:41.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:41.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:41.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:41.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:41.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:41.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:41.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:41.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:41.984184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:41.984703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:41.984703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:41.984703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:46.816243 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:46.816704 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:46.816704 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:46.817713 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:46.817972 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:46.818163 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:46.818310 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:46.818420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:46.818580 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-8f8af46b-411c-4d2c-9cdc-9c2c486aef2a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:46.818809 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:46.818809 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:46.819013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.820757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:46.820757 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.820757 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:46.820757 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:46.820757 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.820757 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:46.820757 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.820757 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:46.821558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.821558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:46.821558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:46.821558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:46.821558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.821558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:46.821558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:46.821558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.821558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:46.821558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:46.821558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:46.821558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.822063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:46.822063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:46.822063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:46.822063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:46.822063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:46.822063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:46.822063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:46.822063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:46.822063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.822063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:46.822063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:46.822063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.822063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:46.822332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.822332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:46.822332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.822332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:46.822332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:46.822332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:46.822332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:46.822332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:46.822332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.822332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:46.822332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:46.822332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.822332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:46.822332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:46.822332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:46.822724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.823079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:46.823079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:46.823079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.823079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:46.823079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.823079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:46.853995 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:46.854622 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:46.855620 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:46.855851 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:34:46.856815 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:46.857360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.858034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:46.858034 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.858034 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:46.858034 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:46.858034 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.858034 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:46.858034 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.858034 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:46.859025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.859025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:46.859025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:46.859025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:46.859025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.859025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:46.859025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:46.859025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:46.859025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.859025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:46.859025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:46.859025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:46.859025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:46.859025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:46.859025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:46.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:46.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:46.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:46.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:46.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:46.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:46.859963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.859963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:46.859963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:46.859963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:46.859963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.859963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:46.859963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:46.859963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:34:46.859963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:46.859963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:46.859963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.859963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:46.860586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:46.860586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:46.860586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.860586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:46.860586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:46.860586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:46.860586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:46.860586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:46.860586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:46.860586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:46.860586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:46.860586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.860586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:46.860586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:46.861267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.861267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:46.861267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.861267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:46.861412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.861412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:46.861412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:46.861412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:46.861412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:46.861412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:46.861412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.861412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:46.861412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:46.861412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.861412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:46.861412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:46.861412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.861412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:46.861412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:46.861412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:46.862188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:46.863337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.863337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:46.863337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:46.863337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.863337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:46.863337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.863337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:46.889582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:46.890024 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:46.890226 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:46.891660 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:46.892863 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:46.893143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.893880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:46.893880 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.893880 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:46.893880 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:46.893880 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.893880 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:46.893880 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.893880 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:46.894956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.894956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:46.894956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:46.894956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:46.894956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.894956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:46.894956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:46.894956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:46.894956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.894956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:46.894956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:46.894956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:46.894956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:46.894956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:46.894956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:46.895242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:46.895242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:46.895242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.895242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:46.895242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:46.895242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.895242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:46.895242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.895242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:46.896412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.896412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:46.896412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:46.896412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:46.896412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.896412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:46.896412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:46.896412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:46.896412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:46.896412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.896412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:46.896660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:46.896660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:46.896660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.896660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:46.896660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:46.896660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:46.896660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:46.896660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:46.896660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:46.896660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:46.896660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:46.896660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.896660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:46.896660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:46.896872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.896872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:46.896872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.896872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.897647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:46.898022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:46.898337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.898337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:46.898337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.898337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:46.929944 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:46.933043 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:46.933043 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:46.933185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:46.933478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:46.933718 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:46.933950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.934435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:46.934435 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.934435 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:46.934435 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:46.934435 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.934435 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:46.934435 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.934435 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:46.935612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.935612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:46.935612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:46.935612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:46.935612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.935612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:46.935612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:46.935612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.935612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:46.935612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:46.935612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:46.935612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:46.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:46.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:46.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:46.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:46.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:46.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:46.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:46.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:46.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:46.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.935906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:46.936107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.936107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:46.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:46.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:46.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:46.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:46.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:46.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:46.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:46.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:46.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:46.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:46.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:46.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:46.940002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.940372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:46.940372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.940372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:46.940660 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:46.944042 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:46.944220 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:46.945223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:46.978149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:46.978332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.978886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:46.978886 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.978886 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:46.978886 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:46.978886 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.978886 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:46.978886 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.978886 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:46.979638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.979638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:46.979638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:46.979638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:46.979638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.979638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:46.979638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:46.979638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.979638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:46.979638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:46.979638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:46.979638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.980012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:46.980012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:46.980012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:46.980012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:46.980012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:46.980012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:46.980012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:46.980012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:46.980012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.980012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:46.980012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:46.980012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.980012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:46.980426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.980426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:46.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:46.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:46.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:46.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:46.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:46.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:46.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:46.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:46.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:46.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:46.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:46.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:46.981028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:46.981416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:46.981416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:46.981416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:51.819103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:51.820438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:51.820633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:51.821616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:51.822047 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:51.822047 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:51.822216 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:51.822295 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:51.822458 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:51.822681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-e88e991e-fb24-4b12-8712-369212ae1041, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:51.822775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.824180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:51.824180 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:51.824180 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:51.824180 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:51.824180 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:51.824180 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:51.824180 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.824180 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:51.824891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.824891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:51.824891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:51.824891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:51.824891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.824891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:51.824891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:51.824891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.824891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:51.824891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:51.824891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:51.824891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.825569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:51.825569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:51.825569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:51.825569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:51.825569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:51.825569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:51.825569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:51.825569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:51.825569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:51.825569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:51.825569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:51.825569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:51.825569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:51.825989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.825989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:51.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:51.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:51.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:51.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:51.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:51.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:51.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:51.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:51.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:51.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:51.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:51.827449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:51.828225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:51.829301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:51.829301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.829301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:51.854687 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:51.857697 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:51.857697 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:51.857697 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:34:51.858078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:51.858362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.859024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:51.859024 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:51.859024 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:51.859024 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:51.859024 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:51.859024 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:51.859024 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.859024 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:51.860440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.860440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:51.860440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:51.860440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:51.860440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.860440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:51.860440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:51.860440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:51.860440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.860440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:51.860440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:51.860440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:51.860440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:51.860440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:51.860440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:51.861559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:51.861559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:51.861559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:51.861559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:51.861559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:51.861559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:51.861559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:51.861559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.861559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:51.861559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.861559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:51.861559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:51.861559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:51.861559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.862332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:51.862332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:51.862332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:34:51.862332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:51.862332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:51.862332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.862332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:51.862332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:51.862332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:51.862332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.862332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:51.862332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:51.862332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:51.863022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:51.863022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:51.863022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:51.863022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:51.863022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:51.863022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:51.863022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:51.863022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:51.863022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:51.863022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:51.863022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.863022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:51.863619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.863619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:51.863619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:51.863619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:51.863619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:51.863619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:51.863619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.863619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:51.863619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:51.863619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.863619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:51.863619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:51.863619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.863619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:51.863619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:51.863619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:51.864563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:51.865367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:51.865367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:51.865367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:51.865367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:51.865367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:51.865367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.865367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:51.890922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:51.893513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:51.894578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:51.894753 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:51.896238 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:51.896468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.897106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:51.897106 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:51.897106 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:51.897106 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:51.897106 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:51.897106 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:51.897106 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.897106 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:51.898116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.898116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:51.898116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:51.898116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:51.898116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.898116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:51.898116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:51.898116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:51.898116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.898116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:51.898116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:51.898116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:51.898116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:51.898116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:51.898116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:51.898894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:51.898894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:51.898894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:51.898894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:51.898894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:51.898894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:51.898894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:51.898894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.898894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:51.899142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.899142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:51.899142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:51.899142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:51.899142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.899142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:51.899142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:51.899142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:51.899142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:51.899142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.899142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:51.899465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:51.899465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:51.899465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.899465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:51.899465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:51.899465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:51.899465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:51.899465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:51.899465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:51.899465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:51.899465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:51.899465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:51.899465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:51.899465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:51.900010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:51.900010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:51.900010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.900010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.900664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:51.901857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:51.902904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:51.902904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:51.902904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.902904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:51.932867 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:51.933879 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:51.934039 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:51.934404 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:51.934714 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:51.934969 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:51.935179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.937812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:51.937812 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:51.937812 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:51.937812 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:51.937812 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:51.937812 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:51.937812 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.937812 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:51.938872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.938872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:51.938872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:51.938872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:51.938872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.938872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:51.938872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:51.938872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.938872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:51.938872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:51.938872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:51.938872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.939111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:51.939111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:51.939111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:51.939111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:51.939111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:51.939111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:51.939111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:51.939111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:51.939111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:51.939111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:51.939111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:51.939111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:51.939111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:51.939310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.939310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:51.940001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.940001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:51.940001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:51.940001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:51.940001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:51.940001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:51.940001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.940001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:51.940001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:51.940001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.940001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:51.940001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:51.940001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.940001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:51.940001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:51.940001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:51.940371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:51.940716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:51.940716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:51.940716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:51.944162 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:51.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:51.945134 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:51.945985 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:52.002080 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:52.003808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:52.003808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:52.003808 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:52.003808 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:52.003808 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:52.003808 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:52.003808 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:52.003808 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:52.003808 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:52.004346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:52.004346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:52.004346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:52.004346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:52.004346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:52.004346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:52.004346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:52.004346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:52.004346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:52.004346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:52.004346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:52.004346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:52.005069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:52.005069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:52.005069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:52.005069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:52.005069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:52.005069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:52.005069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:52.005069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:52.005069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:52.005069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:52.005069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:52.005069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:52.005069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:52.005338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:52.005338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:52.005338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:52.005338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:52.005338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:52.005338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:52.005338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:52.005338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:52.005338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:52.005338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:52.005338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:52.005338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:52.005338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:52.005338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:52.005338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:52.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:52.005973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:52.005973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:52.005973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:52.005973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:52.005973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:52.005973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:56.820573 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:56.821103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:56.821103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:56.823235 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:56.823235 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:56.823235 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:56.823235 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:56.823235 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:56.823451 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:56.823451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.823451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:56.823451 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.823451 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:56.823451 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:56.823451 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.823451 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:56.823451 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.823451 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:56.824418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.824418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:56.824418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:56.824418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:56.824418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.824418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:56.824418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:56.824418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.824418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:56.824418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:56.824418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:56.824418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.824793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:56.824793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:56.824793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:56.824793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:56.824793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:56.824793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:56.824793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:56.824793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:56.824793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.824793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:56.824793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:56.824793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.824793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:56.825111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.825111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:56.825420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.825420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:56.825420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:56.825420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:56.825420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:56.825420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:56.825420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.825420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:56.825420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:56.825420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.825420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:56.825420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:56.825420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.825420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:56.825420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:56.825420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:56.825950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.826405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:56.826405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.826405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:56.857134 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:56.857532 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:56.857661 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:56.857793 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:34:56.858278 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:56.859533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:56.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:56.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:56.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:56.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.859757 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:56.860385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.860385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:56.860385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:56.860385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:56.860385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.860385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:56.860385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:56.860385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:56.860385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.860385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:56.860385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:56.860385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:56.860385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:56.860385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:56.860385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:56.860897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:56.860897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:56.860897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.860897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:56.860897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:56.860897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.860897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:56.860897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.860897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:56.861217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.861217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:56.861217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:56.861217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:56.861217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.861217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:56.861217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:56.861217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:34:56.861217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:56.861217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:56.861217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.861217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:56.861858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:56.861858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:56.861858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.861858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:56.861858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:56.861858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:56.861858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:56.861858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:56.861858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:56.861858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:56.861858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:56.861858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.861858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:56.861858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:56.862349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.862349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:56.862349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.862349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:56.862349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.862349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:56.862349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:56.862349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:56.862349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:56.862349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:56.862349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.862349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:56.862349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:56.862349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:56.862989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:56.863621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:56.863621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:56.863621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:56.863621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:56.863621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:56.863621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.863621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:56.863621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:56.863621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.863621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:56.863621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.863621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:56.892025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:56.892590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:56.892773 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:56.892902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:56.893918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:56.893918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.894033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:56.894033 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.894033 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:56.894033 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:56.894033 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.894033 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:56.894033 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.894033 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:56.894627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.894627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:34:56.894627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:56.894627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:56.894627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.894627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:56.894627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:56.894627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:34:56.894627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.894627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:34:56.894627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:56.894627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:34:56.894627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:56.894627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:34:56.894627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:34:56.895018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:56.895018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:34:56.895018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.895018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:34:56.895018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:34:56.895018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.895018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:34:56.895018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.895018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:34:56.895354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.895354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:56.895354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:56.895354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:56.895354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.895354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:56.895354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:56.895354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:56.895354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:56.895354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.895354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:56.895572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:56.895572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:56.895572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.895572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:56.895572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:56.895572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:56.895572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:56.895572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:56.895572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:56.895572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:56.895572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:56.895572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.895572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:56.895572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:56.895814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.895814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:56.895814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.895814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.896138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:56.896525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:56.896851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.896851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:56.896851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.896851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:56.933903 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:56.934439 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:56.934519 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:56.934735 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:56.935105 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:56.936049 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:56.936049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.936230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:56.936230 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.936230 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:56.936230 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:56.936230 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.936230 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:56.936230 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.936230 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:56.936978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.936978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:56.936978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:56.936978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:56.936978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.936978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:56.936978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:56.936978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.936978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:56.936978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:56.936978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:56.936978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.937259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:56.937259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:56.937259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:56.937259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:56.937259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:56.937259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:56.937259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:56.937259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:56.937259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.937259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:56.937259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:56.937259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.937259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:56.937469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.937469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:56.937685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.937685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:56.937685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:56.937685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:56.937685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:56.937685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:56.937685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.937685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:56.937685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:56.937685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.937685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:56.937685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:56.937685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.937685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:56.937685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:56.937685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:56.937983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.938259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:56.938259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.938259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:34:56.945130 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:34:56.945387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:34:56.945475 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:34:56.946119 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:34:56.993645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:34:56.994224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.994886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:34:56.994886 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.994886 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:34:56.994886 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:34:56.994886 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.994886 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:34:56.994886 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.994886 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:34:56.995914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.049 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.995914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:34:56.995914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:56.995914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:34:56.995914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.995914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:56.995914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:56.995914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.995914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:56.995914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:56.995914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:34:56.995914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.996273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:34:56.996273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:56.996273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:34:56.996273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:56.996273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:34:56.996273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:34:56.996273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:56.996273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:34:56.996273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.996273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:34:56.996273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:34:56.996273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.996273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:34:56.997325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.997325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:34:56.997325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.997325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:34:56.997325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:34:56.997325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:34:56.997325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:34:56.997325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:34:56.997325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.997325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:34:56.997325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:34:56.997325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.997325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:34:56.997325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:34:56.997325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:34:56.998004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:34:56.999079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:34:56.999079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:34:56.999079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:34:56.999079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:34:56.999079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:34:56.999079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:01.821496 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:01.822101 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:01.822101 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:01.823058 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:01.823268 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:01.823342 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:01.823527 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:01.823614 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3268e786-4ff5-48c3-be68-53c3f8dc0f70, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:01.823809 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:01.823809 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:01.824300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.826995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:01.826995 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.826995 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:01.826995 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:01.826995 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.826995 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:01.826995 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.826995 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:01.827959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.827959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:01.827959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:01.827959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:01.827959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.827959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:01.827959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:01.827959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.827959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:01.827959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:01.827959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:01.827959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.828384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:01.828384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:01.828384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:01.828384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:01.828384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:01.828384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:01.828384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:01.828384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:01.828384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.828384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:01.828384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:01.828384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.828384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:01.828804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.828804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:01.829179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.829179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:01.829179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:01.829179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:01.829179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:01.829179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:01.829179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.829179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:01.829179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:01.829179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.829179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:01.829179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:01.829179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.829179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:01.829179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:01.829179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:01.829659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.830228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:01.830228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.830228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:01.857607 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:01.859893 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:01.860056 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:01.860202 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:35:01.860550 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:01.860797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.863768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:01.863768 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.863768 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:01.863768 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:01.863768 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.863768 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:01.863768 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.863768 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:01.864684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.864684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:01.864684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:01.864684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:01.864684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.864684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:01.864684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:01.864684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:01.864684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.864684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:01.864684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:01.864684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:01.864684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:01.864684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:01.864684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:01.865068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:01.865068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:01.865068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.865068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:01.865068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:01.865068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.865068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:01.865068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.865068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:01.865652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.865652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:01.865652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:01.865652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:01.865652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.865652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:01.865652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:01.865652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:35:01.865652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:01.865652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:01.865652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.865652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:01.865941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:01.865941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:01.865941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.865941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:01.865941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:01.865941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:01.865941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:01.865941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:01.865941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:01.865941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:01.865941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:01.865941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.865941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:01.865941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:01.866158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.866158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:01.866158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.866158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:01.869970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.869970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:01.869970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:01.869970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:01.869970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:01.869970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:01.869970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.869970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:01.869970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:01.869970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.869970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:01.869970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:01.869970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.869970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:01.869970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:01.869970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:01.870395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:01.870798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.870798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:01.870798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:01.870798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.870798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:01.870798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.870798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:01.892555 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:01.892880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:01.893018 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:01.893140 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:01.894816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:01.895022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.895476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:01.895476 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.895476 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:01.895476 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:01.895476 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.895476 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:01.895476 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.895476 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:01.896432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.896432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:01.896432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:01.896432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:01.896432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.896432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:01.896432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:01.896432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:01.896432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.896432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:01.896432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:01.896432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:01.896432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:01.896432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:01.896432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:01.897503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:01.897503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:01.897503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.897503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:01.897503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:01.897503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.897503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:01.897503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.897503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:01.897503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.897503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:01.897503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:01.897503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:01.897503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.898115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:01.898115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:01.898115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:01.898115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:01.898115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.898115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:01.898115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:01.898115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:01.898115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.898115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:01.898115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:01.898115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:01.898115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:01.898745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:01.898745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:01.898745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:01.898745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:01.898745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.898745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:01.898745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:01.898745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.898745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:01.898745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.898745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:01.898745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:01.899235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.899960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:01.934574 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:01.934830 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:01.934970 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:01.935186 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:01.937830 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:01.938116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:01.938226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.938731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:01.938731 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.938731 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:01.938731 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:01.938731 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.938731 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:01.938731 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.938731 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:01.939566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.939566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:01.939566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:01.939566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:01.939566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.939566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:01.939566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:01.939566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.939566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:01.939566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:01.939566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:01.939566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.940177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:01.940177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:01.940177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:01.940177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:01.940177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:01.940177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:01.940177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:01.940177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:01.940177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.940177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:01.940177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:01.940177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.940177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:01.940738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.940738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:01.940738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.940738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:01.940738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:01.940738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:01.940738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:01.940738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:01.940738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.940738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:01.940738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:01.940738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.940738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:01.940738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:01.940738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:01.941255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.942050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:01.942050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:01.942050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.942050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:01.942050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.942050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:01.945818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:01.945818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:01.945818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:01.946409 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:01.981207 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:01.981552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.982075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:01.982075 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.982075 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:01.982075 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:01.982075 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.982075 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:01.982075 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.982075 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:01.984095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.984095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:01.984095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:01.984095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:01.984095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.984095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:01.984095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:01.984095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.984095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:01.984095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:01.984095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:01.984095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.984427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:01.984427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:01.984427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:01.984427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:01.984427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:01.984427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:01.984427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:01.984427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:01.984427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.984427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:01.984427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:01.984427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.984427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:01.984654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.984654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:01.984654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.984654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:01.984654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:01.984654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:01.984654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:01.984654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:01.984654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.984654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:01.984654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:01.984654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.984654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:01.984654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:01.984654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:01.984943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:01.985234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:01.985234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:01.985234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:01.985234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:01.985234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:01.985234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:06.822092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:06.824259 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:06.824917 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:06.828285 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:06.828607 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:06.828846 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:06.831104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:06.831372 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:06.831582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:06.831768 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:06.831927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.832998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:06.832998 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:06.832998 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:06.832998 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:06.832998 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:06.832998 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:06.832998 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.832998 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:06.836964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.836964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:06.836964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:06.836964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:06.836964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.836964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:06.836964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:06.836964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.836964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:06.836964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:06.836964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:06.836964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.837249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:06.837249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:06.837249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:06.837249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:06.837249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:06.837249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:06.837249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:06.837249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:06.837249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:06.837249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:06.837249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:06.837249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:06.837249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:06.837490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.837490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:06.837490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.837490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:06.837490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:06.837490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:06.837490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:06.837490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:06.837490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.837490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:06.837490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:06.837490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.837490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:06.837490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:06.837490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:06.837858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:06.838151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:06.838151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:06.838151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:06.838151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:06.838151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.838151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:06.861911 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:06.864673 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:06.864673 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:06.864673 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:35:06.865096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:06.865231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.867585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:06.867585 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:06.867585 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:06.867585 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:06.867585 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:06.867585 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:06.867585 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.867585 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:06.867988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.867988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:06.867988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:06.867988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:06.867988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.867988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:06.867988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:06.867988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:06.867988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.867988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:06.867988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:06.867988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:06.867988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:06.867988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:06.867988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:06.868381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:06.868381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:06.868381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:06.868381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:06.868381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:06.868381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:06.868381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:06.868381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.868381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:06.868829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.868829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:06.868829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:06.868829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:06.868829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.868829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:06.868829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:06.868829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:35:06.868829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:06.868829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:06.868829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.868829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:06.869124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:06.869124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:06.869124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.869124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:06.869124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:06.869124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:06.869124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:06.869124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:06.869124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:06.869124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:06.869124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:06.869124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:06.869124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:06.869124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:06.869475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:06.869475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:06.869475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.869475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:06.872710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.872710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:06.872710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:06.872710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:06.872710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:06.872710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:06.872710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.872710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:06.872710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:06.872710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.872710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:06.872710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:06.872710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.872710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:06.872710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:06.872710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:06.873245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:06.873799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:06.873799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:06.873799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:06.873799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:06.873799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:06.873799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.873799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:06.893019 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:06.893415 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:06.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:06.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:06.894967 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:06.895303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.896503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:06.896503 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:06.896503 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:06.896503 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:06.896503 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:06.896503 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:06.896503 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.896503 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:06.897533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.897533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:06.897533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:06.897533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:06.897533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.897533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:06.897533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:06.897533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:06.897533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.897533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:06.897533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:06.897533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:06.897533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:06.897533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:06.897533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:06.897956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:06.897956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:06.897956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:06.897956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:06.897956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:06.897956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:06.897956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:06.897956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.897956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:06.899573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.899573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:06.899573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:06.899573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:06.899573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.899573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:06.899573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:06.899573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:06.899573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:06.899573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.899573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:06.900027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:06.900027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:06.900027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.900027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:06.900027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:06.900027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:06.900027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:06.900027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:06.900027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:06.900027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:06.900027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:06.900027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:06.900027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:06.900027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:06.900320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:06.900320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:06.900320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.900320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.902032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:06.902944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:06.902944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:06.902944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.902944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:06.936260 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:06.936360 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:06.936410 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:06.936587 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:06.937001 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:06.937296 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:06.937627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:06.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:06.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:06.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:06.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:06.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:06.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:06.939704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.939704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:06.939704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:06.939704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:06.939704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.939704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:06.939704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:06.939704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.939704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:06.939704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:06.939704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:06.939704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.940038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:06.940038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:06.940038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:06.940038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:06.940038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:06.940038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:06.940038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:06.940038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:06.940038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:06.940038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:06.940038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:06.940038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:06.940038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:06.940305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.940305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:06.940815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.940815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:06.940815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:06.940815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:06.940815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:06.940815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:06.940815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.940815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:06.940815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:06.940815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.940815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:06.940815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:06.940815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.940815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:06.940815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:06.940815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:06.941143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:06.941457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:06.941457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:06.941457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:06.946009 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:06.950625 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:06.950625 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:06.950625 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:07.002620 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:07.002919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:07.003521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:07.003521 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:07.003521 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:07.003521 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:07.003521 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:07.003521 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:07.003521 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:07.003521 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:07.004424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:07.004424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:07.004424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:07.004424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:07.004424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:07.004424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:07.004424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:07.004424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:07.004424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:07.004424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:07.004424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:07.004424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:07.004822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:07.004822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:07.004822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:07.004822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:07.004822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:07.004822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:07.004822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:07.004822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:07.004822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:07.004822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:07.004822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:07.004822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:07.004822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:07.005067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:07.005067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:07.005125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:07.005125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:07.005125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:07.005125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:07.005125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:07.005125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:07.005125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:07.005125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:07.005125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:07.005125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:07.005125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:07.005125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:07.005125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:07.005125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:07.005125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:07.005125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:07.005418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:07.005834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:07.005834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:07.005834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:11.826396 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:11.827223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:11.827802 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:11.828606 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:11.829006 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:11.830067 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:11.830205 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:11.831533 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:11.832990 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:11.832990 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:11.832990 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:11.832990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.832990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:11.832990 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:11.832990 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:11.832990 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:11.832990 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:11.832990 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:11.832990 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.832990 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:11.834005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.834005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:11.834005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:11.834005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:11.834005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.834005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:11.834005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:11.834005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.834005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:11.834005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:11.834005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:11.834005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.835081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:11.835081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:11.835081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:11.835081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:11.835081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:11.835081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:11.835081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:11.835081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:11.835081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:11.835081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:11.835081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:11.835081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:11.835081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:11.835683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.835683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:11.835683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.835683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:11.835683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:11.835683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:11.835683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:11.835683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:11.835683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.835683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:11.835683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:11.835683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.835683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:11.835683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:11.835683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:11.836360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:11.837235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:11.837235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:11.837235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:11.837235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:11.837235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.837235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:11.864134 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:11.864476 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:11.864830 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:11.865688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:35:11.866226 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:11.866477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.869448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:11.869448 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:11.869448 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:11.869448 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:11.869448 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:11.869448 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:11.869448 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.869448 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:11.870192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.870192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:11.870192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:11.870192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:11.870192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.870192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:11.870192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:11.870192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:11.870192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.870192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:11.870192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:11.870192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:11.870192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:11.870192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:11.870192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:11.870660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:11.870660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:11.870660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:11.870660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:11.870660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:11.870660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:11.870660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:11.870660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.870660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:11.871116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.871116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:11.871116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:11.871116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:11.871116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.871116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:11.871116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:11.871116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:35:11.871116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:11.871116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:11.871116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.871116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:11.871477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:11.871477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:11.871477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.871477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:11.871477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:11.871477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:11.871477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:11.871477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:11.871477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:11.871477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:11.871477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:11.871477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:11.871477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:11.871477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:11.871875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:11.871875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:11.871875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.871875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:11.872209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.872209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:11.872209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:11.872209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:11.872209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:11.872209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:11.872209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.872209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:11.872209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:11.872209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.872209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:11.872209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:11.872209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.872209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:11.872209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:11.872209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:11.872640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:11.873020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:11.873020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:11.873020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:11.873020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:11.873020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:11.873020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.873020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:11.893608 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:11.896993 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:11.897712 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:11.897712 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:11.898362 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:11.900415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.903417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:11.903417 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:11.903417 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:11.903417 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:11.903417 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:11.903417 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:11.903417 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.903417 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:11.905634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.905634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:11.905634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:11.905634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:11.905634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.905634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:11.905634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:11.905634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:11.905634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.905634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:11.905634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:11.905634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:11.905634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:11.905634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:11.905634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:11.909273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:11.909273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:11.909273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:11.909273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:11.909273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:11.909273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:11.909273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:11.909273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.909273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:11.909273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.909273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:11.909273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:11.909273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:11.909273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.909899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:11.909899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:11.909899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:11.909899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:11.909899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.909899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:11.909899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:11.909899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:11.909899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.909899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:11.909899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:11.909899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:11.909899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:11.910390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:11.910390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:11.910390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:11.910390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:11.910390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:11.910390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:11.910390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:11.910390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:11.910390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:11.910390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.910390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:11.910390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:11.911425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.912010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:11.936600 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:11.936910 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:11.937017 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:11.937190 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:11.939558 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:11.939929 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:11.940233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.941103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:11.941103 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:11.941103 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:11.941103 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:11.941103 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:11.941103 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:11.941103 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.941103 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:11.945049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.945049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:11.945049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:11.945049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:11.945049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.945049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:11.945049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:11.945049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.945049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:11.945049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:11.945049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:11.945049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.945397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:11.945397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:11.945397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:11.945397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:11.945397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:11.945397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:11.945397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:11.945397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:11.945397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:11.945397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:11.945397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:11.945397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:11.945397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:11.945683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.945683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:11.948421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.948421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:11.948421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:11.948421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:11.948421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:11.948421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:11.948421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.948421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:11.948421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:11.948421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.948421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:11.948421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:11.948421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.948421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:11.948421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:11.948421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:11.949127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:11.949740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:11.949740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:11.949740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:11.949740 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:11.950010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:11.950087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:11.952060 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:12.027974 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:12.028344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:12.029024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:12.029024 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:12.029024 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:12.029024 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:12.029024 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:12.029024 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:12.029024 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:12.029024 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:12.030263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.079 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:12.030263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:12.030263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:12.030263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:12.030263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:12.030263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:12.030263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:12.030263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:12.030263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:12.030263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:12.030263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:12.030263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:12.030880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:12.030880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:12.030880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:12.030880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:12.030880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:12.030880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:12.030880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:12.030880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:12.030880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:12.030880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:12.030880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:12.030880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:12.030880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:12.032370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:12.032370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:12.032370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:12.032370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:12.032370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:12.032370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:12.032370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:12.032370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:12.032370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:12.032370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:12.032370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:12.032370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:12.032370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:12.032370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:12.032370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:12.032917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:12.033399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:12.033399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:12.033399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:12.033399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:12.033399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:12.033399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:16.827341 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:16.827879 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:16.827879 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:16.828881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:16.829028 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:16.829143 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:16.829242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:16.829347 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:16.829440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:16.829551 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:16.829660 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:16.829799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.830220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:16.830220 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:16.830220 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:16.830220 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:16.830220 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:16.830220 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:16.830220 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.830220 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:16.830882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.830882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:16.830882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:16.830882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:16.830882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.830882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:16.830882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:16.830882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.830882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:16.830882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:16.830882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:16.830882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.831324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:16.831324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:16.831324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:16.831324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:16.831324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:16.831324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:16.831324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:16.831324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:16.831324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:16.831324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:16.831324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:16.831324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:16.831324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:16.831631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.831631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:16.832212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.832212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:16.832212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:16.832212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:16.832212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:16.832212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:16.832212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.832212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:16.832212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:16.832212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.832212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:16.832212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:16.832212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.832212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:16.832212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:16.832212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:16.832750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:16.833430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:16.833430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.833430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:16.864581 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:16.864978 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:16.865094 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:16.865230 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:35:16.865656 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:16.866079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.866448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:16.866448 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:16.866448 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:16.866448 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:16.866448 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:16.866448 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:16.866448 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.866448 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:16.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:16.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:16.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:16.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:16.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:16.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:16.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:16.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:16.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:16.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:16.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:16.867151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:16.867964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:16.867964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:16.867964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:16.867964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:16.867964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:16.867964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:16.867964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:16.867964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.867964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:16.867964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.867964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:16.867964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:16.867964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:16.867964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.868297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:16.868297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:16.868297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:35:16.868297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:16.868297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:16.868297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.868297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:16.868297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:16.868297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:16.868297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.868297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:16.868297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:16.868297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:16.868633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:16.868633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:16.868633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:16.868633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:16.868633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:16.868633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:16.868633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:16.868633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:16.868633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:16.868633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:16.868633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.868633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:16.868931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.868931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:16.868931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:16.868931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:16.868931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:16.868931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:16.868931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.868931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:16.868931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:16.868931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.868931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:16.868931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:16.868931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.868931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:16.868931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:16.868931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:16.869402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:16.869921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:16.869921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:16.869921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:16.869921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:16.869921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:16.869921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.869921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:16.896052 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:16.896462 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:16.896705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:16.896895 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:16.897626 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:16.899225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.899225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:16.899225 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:16.899225 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:16.899225 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:16.899225 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:16.899225 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:16.899225 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.899225 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:16.900026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.900026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:16.900026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:16.900026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:16.900026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.900026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:16.900026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:16.900026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:16.900026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.900026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:16.900026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:16.900026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:16.900026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:16.900026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:16.900026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:16.900572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:16.900572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:16.900572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:16.900572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:16.900572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:16.900572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:16.900572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:16.900572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.900572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:16.900917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.900917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:16.900917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:16.900917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:16.900917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.900917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:16.900917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:16.900917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:16.900917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:16.900917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.900917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:16.901241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:16.901241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:16.901241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.901241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:16.901241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:16.901241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:16.901241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:16.901241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:16.901241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:16.901241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:16.901241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:16.901241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:16.901241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:16.901241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:16.901844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:16.901844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:16.901844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.901844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:16.901844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.901844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:16.901844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:16.901844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:16.901844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:16.901844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:16.901844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.901844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:16.901844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:16.901844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:16.902310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:16.903020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:16.903020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:16.903020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:16.903020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:16.903020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:16.903020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:16.903020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:16.903020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:16.903020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:16.903020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.903020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:16.937168 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:16.937406 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:16.937532 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:16.937817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:16.938603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:16.938870 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:16.939119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.939687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:16.939687 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:16.939687 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:16.939687 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:16.939687 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:16.939687 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:16.939687 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.939687 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:16.940877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.940877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:16.940877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:16.940877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:16.940877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.940877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:16.940877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:16.940877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.940877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:16.940877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:16.940877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:16.940877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.941270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:16.941270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:16.941270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:16.941270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:16.941270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:16.941270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:16.941270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:16.941270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:16.941270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:16.941270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:16.941270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:16.941270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:16.941270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:16.941639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.941639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:16.943943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.943943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:16.943943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:16.943943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:16.943943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:16.943943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:16.943943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.943943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:16.943943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:16.943943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.943943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:16.943943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:16.943943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.943943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:16.943943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:16.943943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:16.944582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:16.945366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:16.945366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:16.945366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:16.950590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:16.950590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:16.950590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:16.952036 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:17.006431 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:17.006747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:17.007377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:17.007377 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:17.007377 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:17.007377 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:17.007377 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:17.007377 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:17.007377 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:17.007377 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:17.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.057 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:17.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:17.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:17.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:17.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:17.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:17.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:17.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:17.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:17.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:17.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:17.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:17.009019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:17.009019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:17.009019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:17.009019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:17.009019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:17.009019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:17.009019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:17.009019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:17.009019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:17.009019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:17.009019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:17.009019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:17.009019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:17.009308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:17.009308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:17.009910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:17.009910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:17.009910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:17.009910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:17.009910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:17.009910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:17.009910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:17.009910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:17.009910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:17.009910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:17.009910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:17.009910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:17.009910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:17.009910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:17.009910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:17.009910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:17.010349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:17.011484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:17.011484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:17.011484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:21.827984 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:21.828998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:21.828998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:21.830745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:21.831036 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:21.832038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:21.832038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:21.832038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:21.832038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:21.832038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:21.832552 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:21.832552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.832865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:21.832865 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:21.832865 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:21.832865 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:21.832865 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:21.832865 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:21.832865 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.832865 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:21.833911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.833911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:21.833911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:21.833911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:21.833911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.833911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:21.833911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:21.833911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.833911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:21.833911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:21.833911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:21.833911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.834613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:21.834613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:21.834613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:21.834613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:21.834613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:21.834613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:21.834613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:21.834613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:21.834613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:21.834613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:21.834613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:21.834613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:21.834613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:21.835054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.835054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:21.835054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.835054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:21.835054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:21.835054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:21.835054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:21.835054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:21.835054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.835054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:21.835054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:21.835054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.835054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:21.835054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:21.835054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:21.835836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:21.836425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:21.836425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:21.836425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:21.836425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:21.836425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.836425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:21.865040 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:21.865512 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:21.865745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:21.866033 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:35:21.866559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:21.866913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.867684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:21.867684 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:21.867684 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:21.867684 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:21.867684 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:21.867684 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:21.867684 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.867684 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:21.868440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.868440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:21.868440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:21.868440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:21.868440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.868440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:21.868440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:21.868440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:21.868440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.868440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:21.868440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:21.868440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:21.868440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:21.868440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:21.868440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:21.868948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:21.868948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:21.868948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:21.868948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:21.868948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:21.868948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:21.868948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:21.868948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.868948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:21.869227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.869227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:21.869227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:21.869227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:21.869227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.869227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:21.869227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:21.869227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:35:21.869227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:21.869227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:21.869227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.869227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:21.869692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:21.869692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:21.869692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.869692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:21.869692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:21.869692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:21.869692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:21.869692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:21.869692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:21.869692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:21.869692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:21.869692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:21.869692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:21.869692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:21.871089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:21.871089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:21.871089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.871089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:21.871089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.871089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:21.871089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:21.871089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:21.871089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:21.871089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:21.871089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.871089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:21.871089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:21.871089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:21.871577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:21.872031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:21.872031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:21.872031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:21.872031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:21.872031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:21.872031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:21.872031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:21.872031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:21.872031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:21.872031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:21.872031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.872031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:21.896582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:21.897210 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:21.897395 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:21.897551 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:21.898264 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:21.898636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.899397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:21.899397 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:21.899397 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:21.899397 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:21.899397 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:21.899397 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:21.899397 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.899397 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:21.900180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.900180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:21.900180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:21.900180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:21.900180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.900180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:21.900180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:21.900180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:21.900180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.900180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:21.900180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:21.900180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:21.900180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:21.900180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:21.900180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:21.900752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:21.900752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:21.900752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:21.900752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:21.900752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:21.900752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:21.900752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:21.900752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.900752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:21.901175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.901175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:21.901175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:21.901175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:21.901175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.901175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:21.901175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:21.901175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:21.901175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:21.901175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.901175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:21.901591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:21.901591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:21.901591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.901591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:21.901591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:21.901591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:21.901591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:21.901591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:21.901591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:21.901591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:21.901591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:21.901591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:21.901591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:21.901591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:21.901974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:21.901974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:21.901974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.901974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.902351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:21.902775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:21.903054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:21.903054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:21.903054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.903054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:21.937862 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:21.938170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:21.938418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:21.938628 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:21.942072 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:21.943384 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:21.943384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.943951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:21.943951 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:21.943951 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:21.943951 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:21.943951 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:21.943951 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:21.943951 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.943951 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:21.944870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.944870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:21.944870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:21.944870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:21.944870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.944870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:21.944870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:21.944870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.944870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:21.944870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:21.944870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:21.944870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.945336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:21.945336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:21.945336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:21.945336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:21.945336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:21.945336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:21.945336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:21.945336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:21.945336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:21.945336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:21.945336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:21.945336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:21.945336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:21.946133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.946133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:21.946133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.946133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:21.946133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:21.946133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:21.946133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:21.946133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:21.946133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.946133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:21.946133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:21.946133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.946133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:21.946133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:21.946133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:21.946567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:21.947014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:21.947014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:21.947014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:21.947014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:21.947014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:21.947014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:21.950012 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:21.950473 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:21.950733 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:21.951748 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:21.970476 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=242 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:35:22.008509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:22.008509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:22.008509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:22.008509 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:22.008509 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:22.008509 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:22.008509 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:22.008509 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:22.008509 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:22.008509 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:22.009439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:22.009439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:22.009439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:22.009439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:22.009439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:22.009439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:22.009439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:22.009439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:22.009439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:22.009439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:22.009439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:22.009439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:22.010127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:22.010127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:22.010127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:22.010127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:22.010127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:22.010127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:22.010127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:22.010127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:22.010127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:22.010127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:22.010127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:22.010127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:22.010127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:22.010855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:22.010855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:22.010980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:22.010980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:22.010980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:22.010980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:22.010980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:22.010980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:22.010980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:22.010980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:22.010980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:22.010980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:22.010980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:22.010980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:22.010980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:22.010980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:22.010980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:22.010980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:22.011402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:22.012204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:22.012204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:22.012204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:26.828570 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:26.829904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:26.830153 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:26.832790 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:26.833072 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:26.833179 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:26.833366 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:26.833520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:26.833714 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:26.838165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:26.838434 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:26.838503 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:26.838714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.840441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:26.840441 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:26.840441 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:26.840441 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:26.840441 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:26.840441 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:26.840441 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.840441 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:26.841248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.841248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:26.841248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:26.841248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:26.841248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.841248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:26.841248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:26.841248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.841248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:26.841248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:26.841248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:26.841248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.841624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:26.841624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:26.841624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:26.841624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:26.841624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:26.841624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:26.841624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:26.841624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:26.841624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:26.841624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:26.841624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:26.841624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:26.841624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:26.841986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.841986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:26.842052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.842052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:26.842052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:26.842052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:26.842052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:26.842052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:26.842052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.842052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:26.842052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:26.842052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.842052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:26.842052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:26.842052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.842052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:26.842052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:26.842052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:26.842444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:26.842932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:26.842932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.842932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:26.865986 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:26.868129 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:26.868262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:26.868349 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:35:26.869100 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:26.869100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.870030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:26.870030 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:26.870030 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:26.870030 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:26.870030 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:26.870030 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:26.870030 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.870030 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:26.870445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.870445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:26.870445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:26.870445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:26.870445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.870445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:26.870445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:26.870445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:26.870445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.870445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:26.870445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:26.870445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:26.870445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:26.870445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:26.870445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:26.870958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:26.870958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:26.870958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:26.870958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:26.870958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:26.870958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:26.870958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:26.870958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.870958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:26.871181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.871181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:26.871181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:26.871181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:26.871181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.871181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:26.871181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:26.871181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:35:26.871181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:26.871181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:26.871181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.871181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:26.871573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:26.871573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:26.871573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.871573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:26.871573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:26.871573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:26.871573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:26.871573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:26.871573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:26.871573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:26.871573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:26.871573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:26.871573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:26.871573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:26.871916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:26.871916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:26.871916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.871916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:26.872018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.872018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:26.872018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:26.872018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:26.872018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:26.872018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:26.872018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.872018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:26.872018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:26.872018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.872018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:26.872018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:26.872018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.872018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:26.872018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:26.872018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:26.872416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:26.873086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:26.873086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:26.873086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:26.873086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:26.873086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:26.873086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.873086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:26.897312 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:26.897629 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:26.897741 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:26.897920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:26.898348 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:26.898639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.899903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:26.899903 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:26.899903 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:26.899903 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:26.899903 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:26.899903 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:26.899903 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.899903 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:26.900657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.900657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:26.900657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:26.900657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:26.900657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.900657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:26.900657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:26.900657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:26.900657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.900657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:26.900657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:26.900657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:26.900657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:26.900657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:26.900657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:26.901156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:26.901156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:26.901156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:26.901156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:26.901156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:26.901156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:26.901156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:26.901156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.901156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:26.901392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.901392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:26.901392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:26.901392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:26.901392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.901392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:26.901392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:26.901392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:26.901392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:26.901392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.901392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:26.901765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:26.901765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:26.901765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.901765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:26.901765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:26.901765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:26.901765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:26.901765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:26.901765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:26.901765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:26.901765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:26.901765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:26.901765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:26.901765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:26.902083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:26.902083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:26.902083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.902083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.904304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:26.905857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:26.906973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:26.906973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:26.906973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.906973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:26.938296 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:26.938795 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:26.938953 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:26.939132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:26.939424 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:26.939649 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:26.940806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.942015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:26.942015 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:26.942015 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:26.942015 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:26.942015 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:26.942015 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:26.942015 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.942015 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:26.945402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.945402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:26.945402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:26.945402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:26.945402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.945402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:26.945402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:26.945402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.945402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:26.945402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:26.945402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:26.945402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.945713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:26.945713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:26.945713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:26.945713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:26.945713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:26.945713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:26.945713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:26.945713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:26.945713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:26.945713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:26.945713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:26.945713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:26.945713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:26.945941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.945941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:26.946325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.946325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:26.946325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:26.946325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:26.946325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:26.946325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:26.946325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.946325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:26.946325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:26.946325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.946325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:26.946325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:26.946325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.946325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:26.946325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:26.946325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:26.946638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:26.947054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:26.947054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:26.947054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:26.950569 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:26.950848 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:26.950951 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:26.954032 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:27.006966 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:27.008363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:27.008924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:27.008924 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:27.008924 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:27.008924 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:27.008924 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:27.008924 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:27.008924 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:27.008924 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:27.010001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:27.010001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:27.010001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:27.010001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:27.010001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:27.010001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:27.010001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:27.010001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:27.010001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:27.010001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:27.010001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:27.010001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:27.010966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:27.010966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:27.010966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:27.010966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:27.010966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:27.010966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:27.010966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:27.010966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:27.010966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:27.010966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:27.010966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:27.010966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:27.010966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:27.011336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:27.011336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:27.012155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:27.012155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:27.012155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:27.012155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:27.012155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:27.012155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:27.012155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:27.012155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:27.012155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:27.012155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:27.012155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:27.012155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:27.012155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:27.012155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:27.012155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:27.012155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:27.012603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:27.013086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:27.013086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:27.013086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:31.829960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:31.830519 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:31.830519 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:31.834243 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:31.834243 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:31.834243 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:31.834243 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:31.834243 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:31.834991 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:31.834991 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:31.834991 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:31.834991 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:31.834991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.834991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:31.834991 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:31.834991 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:31.834991 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:31.835823 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:31.835823 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:31.835823 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.835823 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:31.835823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.835823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:31.835823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:31.835823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:31.835823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.835823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:31.835823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:31.835823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.835823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:31.836270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:31.836270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:31.836270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.836270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:31.836270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:31.836270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:31.836270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:31.836270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:31.836270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:31.836270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:31.836270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:31.836270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:31.836270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:31.836270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:31.837132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:31.837132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:31.837132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.837132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:31.837708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.837708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:31.837708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:31.837708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:31.837708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:31.837708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:31.837708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.837708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:31.837708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:31.837708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.837708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:31.837708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:31.837708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.837708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:31.837708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:31.837708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:31.838518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:31.839225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:31.839225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.839225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:31.868135 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:31.868495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:31.868688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:31.868853 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:35:31.869503 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:31.870625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.871497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:31.871497 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:31.871497 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:31.871497 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:31.871497 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:31.871497 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:31.871497 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.871497 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:31.872066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.872066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:31.872066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:31.872066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:31.872066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.872066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:31.872066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:31.872066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:31.872066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.872066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:31.872066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:31.872066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:31.872066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:31.872066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:31.872066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:31.872965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:31.872965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:31.872965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:31.872965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:31.872965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:31.872965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:31.872965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:31.872965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.872965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:31.872965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.872965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:31.872965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:31.872965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:31.872965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.873691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:31.873691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:31.873691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:35:31.873691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:31.873691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:31.873691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.873691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:31.873691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:31.873691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:31.873691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.873691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:31.873691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:31.873691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:31.874324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:31.874324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:31.874324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:31.874324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:31.874324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:31.874324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:31.874324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:31.874324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:31.874324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:31.874324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:31.874324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.874324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:31.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:31.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:31.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:31.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:31.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:31.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:31.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:31.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:31.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:31.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:31.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:31.874918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:31.875574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:31.876009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:31.876009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:31.876009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:31.876009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:31.876009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:31.876009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.876009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:31.897941 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:31.898165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:31.898165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:31.898312 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:31.898744 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:31.899455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.900557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:31.900557 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:31.900557 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:31.900557 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:31.900557 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:31.900557 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:31.900557 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.900557 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:31.901166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.901166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:31.901166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:31.901166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:31.901166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.901166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:31.901166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:31.901166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:31.901166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.901166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:31.901166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:31.901166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:31.901166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:31.901166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:31.901166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:31.901782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:31.901782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:31.901782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:31.901782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:31.901782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:31.901782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:31.901782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:31.901782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.901782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:31.902193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.902193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:31.902193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:31.902193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:31.902193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.902193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:31.902193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:31.902193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:31.902193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:31.902193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.902193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:31.902686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:31.902686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:31.902686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.902686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:31.902686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:31.902686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:31.902686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:31.902686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:31.902686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:31.902686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:31.902686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:31.902686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:31.902686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:31.902686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:31.903237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:31.903237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:31.903237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.903237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:31.903237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.903237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:31.903237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:31.903237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:31.903237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:31.903237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:31.903237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.903237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:31.903237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:31.903237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:31.903877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:31.904665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:31.904665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:31.904665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:31.904665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:31.904665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:31.904665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:31.904665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:31.904665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:31.904665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:31.904665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.904665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:31.939883 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:31.942116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:31.942116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:31.942116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:31.942698 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:31.943063 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:31.943448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.944342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:31.944342 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:31.944342 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:31.944342 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:31.944342 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:31.944342 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:31.944342 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.944342 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:31.945450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.945450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:31.945450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:31.945450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:31.945450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.945450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:31.945450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:31.945450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.945450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:31.945450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:31.945450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:31.945450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.946182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:31.946182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:31.946182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:31.946182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:31.946182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:31.946182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:31.946182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:31.946182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:31.946182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:31.946182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:31.946182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:31.946182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:31.946182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:31.946984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.946984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:31.946984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.946984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:31.946984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:31.946984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:31.946984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:31.946984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:31.946984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.946984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:31.946984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:31.946984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.946984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:31.946984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:31.946984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:31.947737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:31.949955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:31.949955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:31.949955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:31.949955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:31.949955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:31.949955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:31.950980 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:31.953023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:31.953023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:31.953023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:32.005520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:32.007067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:32.007268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:32.007268 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:32.007268 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:32.007268 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:32.007268 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:32.007268 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:32.007268 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:32.007268 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:32.008260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:32.008260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:32.008260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:32.008260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:32.008260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:32.008260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:32.008260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:32.008260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:32.008260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:32.008260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:32.008260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:32.008260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:32.009025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:32.009025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:32.009025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:32.009025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:32.009025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:32.009025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:32.009025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:32.009025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:32.009025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:32.009025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:32.009025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:32.009025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:32.009025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:32.009778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:32.009778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:32.009778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:32.009778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:32.009778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:32.009778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:32.009778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:32.009778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:32.009778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:32.009778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:32.009778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:32.009778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:32.009778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:32.009778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:32.009778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:32.010690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:32.011810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:32.011810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:32.011810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:32.011810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:32.011810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:32.011810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:36.831643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:36.834096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:36.834096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:36.836583 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:36.836915 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:36.837155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:36.838968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:36.839517 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:36.842626 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:36.843039 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:36.843294 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:36.843517 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:36.843783 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:36.843783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.846882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:36.846882 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:36.846882 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:36.846882 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:36.846882 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:36.846882 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:36.846882 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.846882 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:36.846882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.011 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.846882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:36.846882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:36.846882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:36.846882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.846882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:36.846882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:36.847528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.847528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:36.847528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:36.847528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:36.847528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.847528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:36.847528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:36.847528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:36.847528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:36.847528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:36.847528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:36.847528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:36.847528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:36.847528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:36.847952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:36.847952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:36.847952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:36.847952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:36.847952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.847952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:36.847952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.847952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:36.847952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:36.847952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:36.847952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:36.847952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:36.847952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.847952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:36.848374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:36.848898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:36.848898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:36.848898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:36.848898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:36.848898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:36.848898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:36.848898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:36.848898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:36.848898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.848898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:36.868824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:36.869632 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:36.869632 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:36.869632 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:35:36.870114 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:36.870406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.871026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:36.871026 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:36.871026 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:36.871026 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:36.871026 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:36.871026 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:36.871026 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.871026 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:36.875571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.875571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:36.875571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:36.875571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:36.875571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.875571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:36.875571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:36.875571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:36.875571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.875571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:36.875571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:36.875571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:36.875571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:36.875571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:36.875571 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:36.876201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:36.876201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:36.876201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:36.876201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:36.876201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:36.876201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:36.876201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:36.876201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.876201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:36.876201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.876201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:36.876201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:36.876201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:36.876201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.876631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:36.876631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:36.876631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:35:36.876631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:36.876631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:36.876631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.876631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:36.876631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:36.876631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:36.876631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.876631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:36.876631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:36.876631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:36.876977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:36.876977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:36.876977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:36.876977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:36.876977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:36.876977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:36.876977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:36.876977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:36.876977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:36.876977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:36.876977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.876977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:36.880818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.880818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:36.880818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:36.880818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:36.880818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:36.880818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:36.880818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.880818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:36.880818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:36.880818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.880818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:36.880818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:36.880818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.880818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:36.880818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:36.880818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:36.881302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:36.881740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:36.881740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:36.881740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:36.881740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:36.881740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:36.881740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.881740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:36.898253 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:36.899335 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:36.899335 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:36.899335 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:36.899838 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:36.900845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.901178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:36.901178 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:36.901178 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:36.901178 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:36.901178 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:36.901178 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:36.901178 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.901178 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:36.902414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.902414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:36.902414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:36.902414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:36.902414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.902414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:36.902414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:36.902414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:36.902414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.902414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:36.902414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:36.902414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:36.902414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:36.902414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:36.902414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:36.902862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:36.902862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:36.902862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:36.902862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:36.902862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:36.902862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:36.902862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:36.902862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.902862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:36.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:36.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:36.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:36.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:36.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:36.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:36.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:36.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:36.908227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:36.908227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:36.908227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.908227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:36.908227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:36.908227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:36.908227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:36.908227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:36.908227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:36.908227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:36.908227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:36.908227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:36.908227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:36.908227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:36.908567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:36.908567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:36.908567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.908567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:36.908567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.908567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:36.908567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:36.908567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:36.908567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:36.908567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:36.908567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.908567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:36.908567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:36.908567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:36.908929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:36.909364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:36.909364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:36.909364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:36.909364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:36.909364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:36.909364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:36.909364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:36.909364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:36.909364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:36.909364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.909364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:36.942013 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:36.943271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:36.943271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:36.943271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:36.943819 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:36.944103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:36.944886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.947437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:36.947437 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:36.947437 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:36.947437 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:36.947437 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:36.947437 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:36.947437 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.947437 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:36.947437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.947437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:36.947437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:36.947437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:36.947437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.947437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:36.947437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:36.947983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.947983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:36.947983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:36.947983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:36.947983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.947983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:36.947983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:36.947983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:36.947983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:36.947983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:36.947983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:36.947983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:36.947983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:36.947983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:36.948379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:36.948379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:36.948379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:36.948379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:36.948379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.948379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:36.948379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.948379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:36.948379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:36.948379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:36.948379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:36.948379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:36.948379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.948379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:36.948798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:36.949314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:36.949314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:36.949314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:36.949314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:36.949314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:36.949314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:36.949314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:36.949314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:36.949314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:36.949314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:36.951789 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:36.952392 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:36.952858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:36.954005 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:37.006963 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:37.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:37.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:37.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:37.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:37.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:37.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:37.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:37.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:37.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:37.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:37.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:37.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:37.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:37.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:37.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:37.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:37.011006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:37.011006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:37.011006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:37.011006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:37.011006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:37.011006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:37.011006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:37.011006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:37.011006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:37.011006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:37.011006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:37.011006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:37.011006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:37.011006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:37.011360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:37.011360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:37.011360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:37.011360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:37.011360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:37.011360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:37.011360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:37.011360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:37.011360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:37.011360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:37.011360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:37.011360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:37.011360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:37.011360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:37.011789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:37.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:37.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:37.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:37.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:37.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:37.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:37.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:37.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:37.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:41.833008 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:41.834697 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:41.834697 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:41.834881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:41.835088 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:41.835275 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:41.835436 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:41.835611 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:41.835766 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:41.835917 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:41.836064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:41.836207 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-66e5230f-43ea-498b-96a2-0a7e462c27c4, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:41.836351 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:41.836494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.838506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:41.838506 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:41.838506 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:41.838506 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:41.838506 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:41.838506 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:41.838506 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.838506 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:41.838506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.838506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:41.838506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:41.838506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:41.838506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.838506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:41.838506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:41.839398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.839398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:41.839398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:41.839398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:41.839398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.839398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:41.839398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:41.839398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:41.839398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:41.839398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:41.839398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:41.839398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:41.839398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:41.839398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:41.840207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:41.840207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:41.840207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:41.840207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:41.840207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.840207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:41.840207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.840207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:41.840207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:41.840207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:41.840207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:41.840207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:41.840207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.840207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:41.840530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:41.841029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:41.841029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:41.841029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:41.841029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:41.841029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:41.841029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:41.841029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:41.841029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:41.841029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.841029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:41.870297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:41.871255 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:41.871255 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:41.871255 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:35:41.871781 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:41.871997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.872637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:41.872637 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:41.872637 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:41.872637 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:41.872637 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:41.872637 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:41.872637 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.872637 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:41.874092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.874092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:41.874092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:41.874092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:41.874092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.874092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:41.874092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:41.874092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:41.874092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.874092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:41.874092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:41.874092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:41.874092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:41.874092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:41.874092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:41.874833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:41.874833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:41.874833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:41.874833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:41.874833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:41.874833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:41.874833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:41.874833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.874833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:41.876036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.876036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:41.876036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:41.876036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:41.876036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.876036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:41.876036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:41.876036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:35:41.876036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:41.876036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:41.876036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.876036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:41.877634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:41.877634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:41.877634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.877634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:41.877634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:41.877634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:41.877634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:41.877634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:41.877634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:41.877634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:41.877634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:41.877634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:41.877634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:41.877634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:41.877961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:41.877961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:41.877961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.877961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:41.878777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.878777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:41.878777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:41.878777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:41.878777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:41.878777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:41.878777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.878777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:41.878777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:41.878777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.878777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:41.878777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:41.878777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.878777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:41.878777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:41.878777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:41.879759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:41.880796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:41.880796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:41.880796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:41.880796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:41.880796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:41.880796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.880796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:41.898907 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:41.899871 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:41.899871 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:41.899871 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:41.900044 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:41.900293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.903888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:41.904380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:41.904380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:41.904380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:41.904380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:41.904380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:41.904380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:41.904380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:41.904380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:41.904380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:41.904380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:41.904380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:41.904380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:41.904380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.904380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:41.904380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.904770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:41.904770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:41.904770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:41.904770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.904770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:41.904770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:41.904770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:41.904770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:41.904770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.904770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:41.904770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:41.904770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:41.904770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.905153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:41.905153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:41.905153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:41.905153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:41.905153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:41.905153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:41.905153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:41.905153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:41.905153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:41.905153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:41.905153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:41.905153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:41.905153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:41.905435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.905435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.906166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:41.906863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:41.907282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:41.907282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:41.907282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.907282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:41.942624 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:41.943718 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:41.943718 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:41.943718 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:41.944055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:41.944386 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:41.944861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.948245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:41.948245 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:41.948245 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:41.948245 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:41.948245 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:41.948245 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:41.948245 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.948245 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:41.948245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.948245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:41.948245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:41.948245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:41.948245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.948245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:41.948245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:41.948651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.948651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:41.948651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:41.948651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:41.948651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.948651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:41.948651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:41.948651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:41.948651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:41.948651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:41.948651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:41.948651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:41.948651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:41.948651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:41.948972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:41.948972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:41.948972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:41.948972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:41.948972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.948972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:41.948972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.948972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:41.948972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:41.948972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:41.948972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:41.948972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:41.948972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.948972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:41.949205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:41.949531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:41.949531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:41.949531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:41.949531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:41.949531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:41.949531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:41.949531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:41.949531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:41.949531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:41.949531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:41.952303 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:41.954688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:41.954688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:41.954688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:42.010318 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:42.010575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:42.013589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:42.013589 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:42.013589 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:42.013589 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:42.013589 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:42.013589 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:42.013589 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:42.013589 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:42.013589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:42.013589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:42.013589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:42.013589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:42.013589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:42.013589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:42.013589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:42.014162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:42.014162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:42.014162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:42.014162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:42.014162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:42.014162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:42.014162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:42.014162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:42.014162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:42.014162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:42.014162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:42.014162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:42.014162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:42.014162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:42.014501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:42.014501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:42.014501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:42.014501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:42.014501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:42.014501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:42.014501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:42.014501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:42.014501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:42.014501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:42.014501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:42.014501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:42.014501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:42.014501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:42.015047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:42.015593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:42.015593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:42.015593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:42.015593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:42.015593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:42.015593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:42.015593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:42.015593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:42.015593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:46.834031 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:46.834479 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:46.834479 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:46.837659 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:46.837659 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:46.837659 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:46.837659 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:46.837659 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:46.837846 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:46.837846 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:46.837846 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:46.837846 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:46.837846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.837846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:46.837846 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:46.837846 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:46.837846 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:46.838159 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:46.838159 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:46.838159 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.838159 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:46.838159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.838159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:46.838159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:46.838159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:46.838159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.838159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:46.838159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:46.838159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.838159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:46.838496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:46.838496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:46.838496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.838496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:46.838496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:46.838496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:46.838496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:46.838496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:46.838496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:46.838496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:46.838496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:46.838496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:46.838496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:46.838496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:46.838834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:46.838834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:46.838834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.838834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:46.838909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.838909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:46.838909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:46.838909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:46.838909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:46.838909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:46.838909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.838909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:46.838909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:46.838909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.838909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:46.838909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:46.838909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.838909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:46.838909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:46.838909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:46.839221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:46.839617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:46.839617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.839617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:46.870864 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:46.871205 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:46.871403 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:46.871616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:35:46.872023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:46.872297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.872885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:46.872885 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:46.872885 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:46.872885 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:46.872885 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:46.872885 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:46.872885 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.872885 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:46.873786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.873786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:46.873786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:46.873786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:46.873786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.873786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:46.873786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:46.873786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:46.873786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.873786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:46.873786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:46.873786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:46.873786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:46.873786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:46.873786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:46.874249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:46.874249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:46.874249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:46.874249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:46.874249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:46.874249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:46.874249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:46.874249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.874249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:46.874744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.874744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:46.874744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:46.874744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:46.874744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.874744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:46.874744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:46.874744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:35:46.874744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:46.874744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:46.874744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.874744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:46.875052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:46.875052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:46.875052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.875052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:46.875052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:46.875052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:46.875052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:46.875052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:46.875052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:46.875052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:46.875052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:46.875052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:46.875052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:46.875052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:46.875326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:46.875326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:46.875326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.875326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:46.875870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.875870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:46.875870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:46.875870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:46.875870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:46.875870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:46.875870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.875870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:46.875870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:46.875870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.875870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:46.875870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:46.875870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.875870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:46.875870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:46.875870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:46.876161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:46.880919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:46.880919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:46.880919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:46.880919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:46.880919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:46.880919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.880919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:46.900923 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:46.901272 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:46.901448 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:46.901782 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:46.902094 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:46.902389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.903199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:46.903199 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:46.903199 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:46.903199 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:46.903199 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:46.903199 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:46.903199 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.903199 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:46.903840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.903840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:46.903840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:46.903840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:46.903840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.903840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:46.903840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:46.903840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:46.903840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.903840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:46.903840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:46.903840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:46.903840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:46.903840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:46.903840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:46.904171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:46.904171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:46.904171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:46.904171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:46.904171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:46.904171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:46.904171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:46.904171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.904171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:46.904928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.904928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:46.904928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:46.904928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:46.904928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.904928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:46.904928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:46.904928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:46.904928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:46.904928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.904928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:46.905194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:46.905194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:46.905194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.905194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:46.905194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:46.905194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:46.905194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:46.905194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:46.905194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:46.905194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:46.905194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:46.905194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:46.905194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:46.905194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:46.905449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:46.905449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:46.905449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.905449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.905998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:46.906323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:46.906750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:46.906750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:46.906750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.906750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:46.950209 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:46.950639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:46.950827 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:46.951019 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:46.951525 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:46.951767 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:46.952021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.952421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:46.952421 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:46.952421 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:46.952421 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:46.952421 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:46.952421 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:46.952421 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.952421 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:46.953360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.953360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:46.953360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:46.953360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:46.953360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.953360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:46.953360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:46.953360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.953360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:46.953360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:46.953360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:46.953360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.953740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:46.953740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:46.953740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:46.953740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:46.953740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:46.953740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:46.953740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:46.953740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:46.953740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:46.953740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:46.953740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:46.953740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:46.953740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:46.954063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.954063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:46.954063 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:46.954450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.954450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:46.954450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:46.954450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:46.954450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:46.954450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:46.954450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.954450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:46.954450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:46.954450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.954450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:46.954450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:46.954450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.954450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:46.954450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:46.954450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:46.954934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:46.955426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:46.955426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:46.955426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:46.955426 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:46.955426 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:46.955659 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:47.014059 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:47.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:47.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:47.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:47.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:47.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:47.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:47.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:47.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:47.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:47.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.060 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:47.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:47.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:47.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:47.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:47.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:47.018094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:47.018094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:47.018094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:47.018094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:47.018094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:47.018094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:47.018094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:47.018094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:47.018094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:47.018094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:47.018094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:47.018094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:47.018094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:47.018094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:47.018526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:47.018526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:47.018526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:47.018526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:47.018526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:47.018526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:47.018526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:47.018526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:47.018526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:47.018526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:47.018526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:47.018526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:47.018526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:47.018943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:47.019782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:47.019782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:47.019782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:47.019782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:47.019782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:47.019782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:47.019782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:47.019782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:47.019782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:47.019782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:47.019782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:47.019782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:51.834532 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:51.835194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:51.835194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:51.835828 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:51.836010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:51.836157 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:51.836311 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:51.836533 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:51.836599 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:51.836760 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:51.836897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:51.837032 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:51.837163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.837729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:51.837729 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:51.837729 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:51.837729 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:51.837729 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:51.837729 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:51.837729 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.837729 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:51.838395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.838395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:51.838395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:51.838395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:51.838395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.838395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:51.838395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:51.838395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.838395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:51.838395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:51.838395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:51.838395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.838796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:51.838796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:51.838796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:51.838796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:51.838796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:51.838796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:51.838796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:51.838796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:51.838796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:51.838796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:51.838796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:51.838796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:51.838796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:51.839152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.839152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:51.839152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.839152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:51.839152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:51.839152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:51.839152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:51.839152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:51.839152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.839152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:51.839152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:51.839152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.839152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:51.839152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:51.839152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:51.839584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:51.840186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:51.840186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:51.840186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:51.840186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:51.840186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.840186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:51.871293 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:51.871530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:51.871653 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:51.871761 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:35:51.872093 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:51.872242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.872648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:51.872648 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:51.872648 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:51.872648 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:51.872648 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:51.872648 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:51.872648 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.872648 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:51.873300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.873300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:51.873300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:51.873300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:51.873300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.873300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:51.873300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:51.873300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:51.873300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.873300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:51.873300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:51.873300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:51.873300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:51.873300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:51.873300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:51.873787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:51.873787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:51.873787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:51.873787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:51.873787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:51.873787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:51.873787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:51.873787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.873787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:51.874027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.874027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:51.874027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:51.874027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:51.874027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.874027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:51.874027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:51.874027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:35:51.874027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:51.874027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:51.874027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.874027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:51.874354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:51.874354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:51.874354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.874354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:51.874354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:51.874354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:51.874354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:51.874354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:51.874354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:51.874354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:51.874354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:51.874354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:51.874354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:51.874354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:51.875080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:51.875080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:51.875080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.875080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:51.875080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.875080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:51.875080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:51.875080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:51.875080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:51.875080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:51.875080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.875080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:51.875080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:51.875080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:51.875464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:51.876015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:51.876015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:51.876015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:51.876015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:51.876015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:51.876015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:51.876015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:51.876015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:51.876015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:51.876015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:51.876015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.876015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:51.902820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:51.903080 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:51.903174 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:51.903280 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:51.903706 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:51.904030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.904550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:51.904550 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:51.904550 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:51.904550 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:51.904550 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:51.904550 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:51.904550 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.904550 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:51.905428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.905428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:51.905428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:51.905428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:51.905428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.905428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:51.905428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:51.905428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:51.905428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.905428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:51.905428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:51.905428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:51.905428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:51.905428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:51.905428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:51.905863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:51.905863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:51.905863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:51.905863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:51.905863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:51.905863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:51.905863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:51.905863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.905863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:51.906315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.906315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:51.906315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:51.906315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:51.906315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.906315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:51.906315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:51.906315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:51.906315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:51.906315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.906315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:51.906657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:51.906657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:51.906657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.906657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:51.906657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:51.906657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:51.906657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:51.906657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:51.906657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:51.906657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:51.906657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:51.906657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:51.906657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:51.906657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:51.907024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:51.907024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:51.907024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.907024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.907439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:51.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:51.908373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:51.908373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:51.908373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.908373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:51.950672 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:51.950990 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:51.951161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:51.951370 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:51.951827 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:51.952053 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:51.952326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.952882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:51.952882 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:51.952882 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:51.952882 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:51.952882 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:51.952882 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:51.952882 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.952882 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:51.953760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.953760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:51.953760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:51.953760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:51.953760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.953760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:51.953760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:51.953760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.953760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:51.953760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:51.953760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:51.953760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.954077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:51.954077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:51.954077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:51.954077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:51.954077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:51.954077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:51.954077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:51.954077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:51.954077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:51.954077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:51.954077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:51.954077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:51.954077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:51.954286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.954286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:51.954497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.954497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:51.954497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:51.954497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:51.954497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:51.954497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:51.954497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.954497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:51.954497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:51.954497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.954497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:51.954497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:51.954497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.954497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:51.954497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:51.954497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:51.954999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:51.955477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:51.955477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:51.955477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:51.955477 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:51.955997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:51.957658 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:51.958476 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:52.011456 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:52.011798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:52.012350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:52.012350 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:52.012350 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:52.012350 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:52.012350 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:52.012350 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:52.012350 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:52.012350 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:52.014587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:52.014587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:52.014587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:52.014587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:52.014587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:52.014587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:52.014587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:52.014587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:52.014587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:52.014587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:52.014587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:52.014587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:52.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:52.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:52.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:52.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:52.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:52.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:52.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:52.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:52.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:52.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:52.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:52.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:52.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:52.015202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:52.015202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:52.015202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:52.015202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:52.015202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:52.015202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:52.015202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:52.015202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:52.015202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:52.015202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:52.015202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:52.015202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:52.015202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:52.015202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:52.015202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:52.015456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:52.015784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:52.015784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:52.015784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:52.015784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:52.015784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:52.015784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:56.834904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:56.835503 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:56.835503 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:56.837744 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:56.837744 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:56.837744 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:56.837744 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:56.837744 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:56.837982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:56.837982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:56.838112 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:56.838222 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:56.838332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.839576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:56.839576 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:56.839576 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:56.839576 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:56.839576 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:56.839576 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:56.839576 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.839576 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:56.840592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.840592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:56.840592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:56.840592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:56.840592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.840592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:56.840592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:56.840592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.840592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:56.840592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:56.840592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:56.840592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.841004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:56.841004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:56.841004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:56.841004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:56.841004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:56.841004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:56.841004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:56.841004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:56.841004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:56.841004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:56.841004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:56.841004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:56.841004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:56.841404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.841404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:56.841641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.841641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:56.841641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:56.841641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:56.841641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:56.841641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:56.841641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.841641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:56.841641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:56.841641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.841641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:56.841641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:56.841641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.841641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:56.841641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:56.841641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:56.842200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:56.842772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:56.842772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.842772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:56.873292 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:56.873612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:56.873766 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:56.873906 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:35:56.874248 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:56.874455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.876782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:56.876782 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:56.876782 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:56.876782 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:56.876782 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:56.876782 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:56.876782 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.876782 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:56.877652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.877652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:56.877652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:56.877652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:56.877652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.877652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:56.877652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:56.877652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:56.877652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.877652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:56.877652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:56.877652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:56.877652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:56.877652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:56.877652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:56.878230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:56.878230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:56.878230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:56.878230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:56.878230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:56.878230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:56.878230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:56.878230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.878230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:56.878500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.878500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:56.878500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:56.878500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:56.878500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.878500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:56.878500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:56.878500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:35:56.878500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:56.878500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:56.878500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.878500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:56.878814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:56.878814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:56.878814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.878814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:56.878814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:56.878814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:56.878814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:56.878814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:56.878814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:56.878814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:56.878814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:56.878814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:56.878814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:56.878814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:56.879191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:56.879191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:56.879191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.879191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:56.880210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.880210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:56.880210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:56.880210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:56.880210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:56.880210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:56.880210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.880210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:56.880210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:56.880210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.880210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:56.880210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:56.880210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.880210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:56.880210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:56.880210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:56.880506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:56.880976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:56.880976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:56.880976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:56.880976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:56.880976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:56.880976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.880976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:56.903249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:56.903634 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:56.903780 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:56.903922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:56.904335 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:56.904611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.905193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:56.905193 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:56.905193 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:56.905193 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:56.905193 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:56.905193 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:56.905193 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.905193 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:56.905901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.905901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:35:56.905901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:56.905901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:56.905901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.905901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:56.905901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:56.905901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:35:56.905901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.905901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:35:56.905901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:56.905901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:35:56.905901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:56.905901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:35:56.905901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:35:56.906190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:56.906190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:35:56.906190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:56.906190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:35:56.906190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:35:56.906190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:56.906190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:35:56.906190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.906190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:35:56.906812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.906812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:56.906812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:56.906812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:56.906812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.906812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:56.906812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:56.906812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:56.906812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:56.906812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.906812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:56.907079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:56.907079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:56.907079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.907079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:56.907079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:56.907079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:56.907079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:56.907079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:56.907079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:56.907079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:56.907079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:56.907079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:56.907079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:56.907079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:56.907370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:56.907370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:56.907370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.907370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.908008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:56.908733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:56.909133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:56.909133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:56.909133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.909133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:56.951164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:56.951420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:56.951508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:56.951715 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:56.952078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:56.952382 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:56.952863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.953400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:56.953400 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:56.953400 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:56.953400 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:56.953400 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:56.953400 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:56.953400 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.953400 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:56.954308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.954308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:56.954308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:56.954308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:56.954308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.954308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:56.954308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:56.954308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.954308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:56.954308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:56.954308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:56.954308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.954558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:56.954558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:56.954558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:56.954558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:56.954558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:56.954558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:56.954558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:56.954558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:56.954558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:56.954558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:56.954558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:56.954558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:56.954558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:56.954856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.954856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:56.955556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.955556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:56.955556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:56.955556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:56.955556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:56.955556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:56.955556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.955556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:56.955556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:56.955556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.955556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:56.955556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:56.955556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.955556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:56.955556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:56.955556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:56.955922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:56.956344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:56.956344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:56.956344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:35:56.956344 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:35:56.956440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:35:56.956597 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:35:56.957499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:35:57.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:35:57.003945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:57.004560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:35:57.004560 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:57.004560 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:35:57.004560 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:35:57.004560 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:57.004560 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:35:57.004560 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:57.004560 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:35:57.005774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:57.005774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:35:57.005774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:57.005774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:35:57.005774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:57.005774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:57.005774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:57.005774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:57.005774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:57.005774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:57.005774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:35:57.005774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:57.006165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:35:57.006165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:57.006165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:35:57.006165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:57.006165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:35:57.006165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:35:57.006165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:57.006165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:35:57.006165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:57.006165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:35:57.006165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:35:57.006165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:57.006165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:35:57.006500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:57.006500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:35:57.007471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:57.007471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:35:57.007471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:35:57.007471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:35:57.007471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:35:57.007471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:35:57.007471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:57.007471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:35:57.007471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:35:57.007471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:57.007471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:35:57.007471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:35:57.007471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:35:57.007471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:35:57.007471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:35:57.007471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:35:57.008044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:35:57.008585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:35:57.008585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:35:57.008585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:01.835325 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:01.835800 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:01.835911 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:01.837007 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:01.837299 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:01.837529 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:01.837771 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:01.837951 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:01.838126 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:01.838299 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:01.838451 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:01.843265 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:01.845226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.846806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:01.846806 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:01.846806 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:01.846806 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:01.846806 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:01.846806 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:01.846806 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.846806 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:01.847934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.011 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.847934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:01.847934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:01.847934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:01.847934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.847934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:01.847934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:01.847934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.847934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:01.847934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:01.847934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:01.847934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.848221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:01.848221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:01.848221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:01.848221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:01.848221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:01.848221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:01.848221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:01.848221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:01.848221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:01.848221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:01.848221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:01.848221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:01.848221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:01.848443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.848443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:01.849085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.849085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:01.849085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:01.849085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:01.849085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:01.849085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:01.849085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.849085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:01.849085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:01.849085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.849085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:01.849085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:01.849085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.849085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:01.849085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:01.849085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:01.849410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:01.849803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:01.849803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.849803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:01.873813 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:01.874729 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:01.874879 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:01.874928 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:36:01.877125 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:01.877318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.877848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:01.877848 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:01.877848 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:01.877848 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:01.877848 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:01.877848 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:01.877848 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.877848 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:01.878452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.878452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:01.878452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:01.878452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:01.878452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.878452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:01.878452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:01.878452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:01.878452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.878452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:01.878452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:01.878452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:01.878452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:01.878452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:01.878452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:01.878758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:01.878758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:01.878758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:01.878758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:01.878758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:01.878758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:01.878758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:01.878758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.878758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:01.879120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.879120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:01.879120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:01.879120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:01.879120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.879120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:01.879120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:01.879120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:36:01.879120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:01.879120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:01.879120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.879120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:01.879366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:01.879366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:01.879366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.879366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:01.879366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:01.879366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:01.879366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:01.879366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:01.879366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:01.879366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:01.879366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:01.879366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:01.879366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:01.879366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:01.879598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:01.879598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:01.879598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.879598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:01.879849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.879849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:01.879849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:01.879849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:01.879849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:01.879849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:01.879849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.879849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:01.879849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:01.879849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.879849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:01.879849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:01.879849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.879849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:01.879849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:01.879849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:01.880167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:01.880608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:01.880608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:01.880608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:01.880608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:01.880608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:01.880608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.880608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:01.903679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:01.904008 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:01.904140 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:01.904282 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:01.904934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:01.905147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.905656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:01.905656 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:01.905656 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:01.905656 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:01.905656 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:01.905656 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:01.905656 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.905656 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:01.906601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.906601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:01.906601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:01.906601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:01.906601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.906601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:01.906601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:01.906601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:01.906601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.906601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:01.906601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:01.906601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:01.906601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:01.906601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:01.906601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:01.906945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:01.906945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:01.906945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:01.906945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:01.906945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:01.906945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:01.906945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:01.906945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.906945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:01.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:01.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:01.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:01.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:01.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:01.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:01.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:01.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:01.907868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:01.907868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:01.907868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.907868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:01.907868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:01.907868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:01.907868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:01.907868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:01.907868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:01.907868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:01.907868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:01.907868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:01.907868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:01.907868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:01.908128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:01.908128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:01.908128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.908128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.911471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:01.912157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:01.912511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:01.912511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:01.912511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.912511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:01.951638 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:01.952214 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:01.952389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:01.952611 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:01.956131 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:01.956438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:01.956816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.957227 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:01.962151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:01.962151 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:01.962151 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:01.962151 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:01.962151 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:01.962151 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:01.962151 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.962151 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:01.963005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.963005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:01.963005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:01.963005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:01.963005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.963005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:01.963005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:01.963005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.963005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:01.963005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:01.963005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:01.963005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.963371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:01.963371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:01.963371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:01.963371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:01.963371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:01.963371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:01.963371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:01.963371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:01.963371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:01.963371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:01.963371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:01.963371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:01.963371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:01.963882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.963882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:01.963882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.963882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:01.963882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:01.963882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:01.963882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:01.963882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:01.963882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.963882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:01.963882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:01.963882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.963882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:01.963882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:01.963882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:01.964262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:01.964750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:01.964750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:01.964750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:01.964750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:01.964750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.964750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:01.964750 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:01.964750 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:01.964750 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:01.997816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:01.998027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.998593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:01.998593 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:01.998593 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:01.998593 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:01.998593 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:01.998593 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:01.998593 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.998593 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:01.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:01.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:01.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:01.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:01.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:01.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:01.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:01.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:01.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:01.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:01.999586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:01.999586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:01.999586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:01.999586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:01.999586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:01.999586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:01.999586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:01.999586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:01.999586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:01.999586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:01.999586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:01.999586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:01.999586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:01.999829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:01.999829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:02.000087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:02.000087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:02.000087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:02.000087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:02.000087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:02.000087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:02.000087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:02.000087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:02.000087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:02.000087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:02.000087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:02.000087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:02.000087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:02.000087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:02.000087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:02.000087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:02.000396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:02.000756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:02.000756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:02.000756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:06.835761 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:06.836249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:06.836249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:06.838248 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:06.838443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:06.838547 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:06.838680 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:06.839061 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:06.839245 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:06.839365 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:06.839470 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:06.840752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:06.840752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.842349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:06.842349 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:06.842349 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:06.842349 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:06.842349 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:06.842349 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:06.842349 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.842349 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:06.843246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.843246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:06.843246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:06.843246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:06.843246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.843246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:06.843246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:06.843246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.843246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:06.843246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:06.843246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:06.843246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.843646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:06.843646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:06.843646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:06.843646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:06.843646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:06.843646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:06.843646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:06.843646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:06.843646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:06.843646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:06.843646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:06.843646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:06.843646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:06.844034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.844034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:06.845325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.845325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:06.845325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:06.845325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:06.845325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:06.845325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:06.845325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.845325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:06.845325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:06.845325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.845325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:06.845325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:06.845325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.845325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:06.845325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:06.845325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:06.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:06.846451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:06.846451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.846451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:06.874771 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:06.875116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:06.875279 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:06.875460 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:36:06.875998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:06.876194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.876805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:06.876805 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:06.876805 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:06.876805 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:06.876805 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:06.876805 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:06.876805 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.876805 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:06.877570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.877570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:06.877570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:06.877570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:06.877570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.877570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:06.877570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:06.877570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:06.877570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.877570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:06.877570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:06.877570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:06.877570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:06.877570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:06.877570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:06.878057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:06.878057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:06.878057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:06.878057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:06.878057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:06.878057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:06.878057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:06.878057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.878057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:06.878763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.878763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:06.878763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:06.878763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:06.878763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.878763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:06.878763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:06.878763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:36:06.878763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:06.878763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:06.878763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.878763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:06.879140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:06.879140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:06.879140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.879140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:06.879140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:06.879140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:06.879140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:06.879140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:06.879140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:06.879140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:06.879140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:06.879140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:06.879140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:06.879140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:06.879722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:06.879722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:06.879722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.879722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:06.879722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.879722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:06.879722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:06.879722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:06.879722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:06.879722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:06.879722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.879722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:06.879722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:06.879722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:06.880259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:06.880885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:06.880885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:06.880885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:06.880885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:06.880885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:06.880885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:06.880885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:06.880885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:06.880885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:06.880885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:06.880885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.880885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:06.904151 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:06.904496 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:06.904701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:06.904876 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:06.905572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:06.905698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.906253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:06.906253 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:06.906253 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:06.906253 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:06.906253 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:06.906253 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:06.906253 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.906253 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:06.907246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.907246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:06.907246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:06.907246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:06.907246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.907246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:06.907246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:06.907246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:06.907246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.907246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:06.907246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:06.907246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:06.907246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:06.907246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:06.907246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:06.907661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:06.907661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:06.907661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:06.907661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:06.907661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:06.907661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:06.907661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:06.907661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.907661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:06.908241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.908241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:06.908241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:06.908241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:06.908241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.908241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:06.908241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:06.908241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:06.908241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:06.908241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.908241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:06.908502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:06.908502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:06.908502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.908502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:06.908502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:06.908502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:06.908502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:06.908502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:06.908502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:06.908502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:06.908502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:06.908502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:06.908502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:06.908502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:06.908771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:06.908771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:06.908771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.908771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.909511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:06.909980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:06.910463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:06.910463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:06.910463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.910463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:06.952989 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:06.952989 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:06.952989 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:06.952989 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:06.953287 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:06.953503 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:06.953763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.955735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:06.955735 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:06.955735 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:06.955735 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:06.955735 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:06.955735 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:06.955735 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.955735 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:06.956788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.956788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:06.956788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:06.956788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:06.956788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.956788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:06.956788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:06.956788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.956788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:06.956788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:06.956788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:06.956788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.957205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:06.957205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:06.957205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:06.957205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:06.957205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:06.957205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:06.957205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:06.957205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:06.957205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:06.957205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:06.957205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:06.957205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:06.957205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:06.957557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.957557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:06.958797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.958797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:06.958797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:06.958797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:06.958797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:06.958797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:06.958797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.958797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:06.958797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:06.958797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.958797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:06.958797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:06.958797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.958797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:06.958797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:06.958797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:06.959289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:06.959788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:06.959788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:06.959788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:06.964273 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:06.964550 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:06.964696 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:06.965446 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:07.021961 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:07.022159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:07.022614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:07.022614 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:07.022614 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:07.022614 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:07.022614 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:07.022614 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:07.022614 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:07.022614 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:07.023327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:07.023327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:07.023327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:07.023327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:07.023327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:07.023327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:07.023327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:07.023327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:07.023327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:07.023327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:07.023327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:07.023327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:07.023615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:07.023615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:07.023615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:07.023615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:07.023615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:07.023615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:07.023615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:07.023615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:07.023615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:07.023615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:07.023615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:07.023615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:07.023615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:07.023865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:07.023865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:07.024628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:07.024628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:07.024628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:07.024628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:07.024628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:07.024628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:07.024628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:07.024628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:07.024628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:07.024628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:07.024628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:07.024628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:07.024628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:07.024628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:07.024628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:07.024628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:07.024893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:07.025216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:07.025216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:07.025216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:11.836209 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:11.836818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:11.836818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:11.839041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:11.839360 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:11.839522 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:11.839678 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:11.839807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:11.839920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:11.840026 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:11.840128 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:11.840228 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:11.840328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:11.840429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.841242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:11.841242 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:11.841242 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:11.841242 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:11.841242 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:11.841242 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:11.841242 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.841242 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:11.841976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.841976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:11.841976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:11.841976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:11.841976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.841976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:11.841976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:11.841976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.841976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:11.841976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:11.841976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:11.841976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.842284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:11.842284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:11.842284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:11.842284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:11.842284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:11.842284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:11.842284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:11.842284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:11.842284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:11.842284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:11.842284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:11.842284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:11.842284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:11.842530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.842530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:11.842751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.842751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:11.842751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:11.842751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:11.842751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:11.842751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:11.842751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.842751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:11.842751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:11.842751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.842751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:11.842751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:11.842751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.842751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:11.842751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:11.842751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:11.843075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:11.843450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:11.843450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.843450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:11.875191 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:11.875520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:11.876411 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:11.876629 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:36:11.877460 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:11.877685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.881188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:11.881188 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:11.881188 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:11.881188 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:11.881188 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:11.881188 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:11.881188 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.881188 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:11.881950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.881950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:11.881950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:11.881950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:11.881950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.881950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:11.881950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:11.881950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:11.881950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.881950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:11.881950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:11.881950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:11.881950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:11.881950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:11.881950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:11.882448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:11.882448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:11.882448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:11.882448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:11.882448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:11.882448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:11.882448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:11.882448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.882448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:11.882777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.882777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:11.882777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:11.882777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:11.882777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.882777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:11.882777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:11.882777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:36:11.882777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:11.882777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:11.882777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.882777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:11.883140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:11.883140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:11.883140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.883140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:11.883140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:11.883140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:11.883140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:11.883140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:11.883140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:11.883140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:11.883140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:11.883140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:11.883140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:11.883140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:11.883552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:11.883552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:11.883552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.883552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:11.883686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.883686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:11.883686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:11.883686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:11.883686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:11.883686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:11.883686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.883686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:11.883686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:11.883686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.883686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:11.883686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:11.883686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.883686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:11.883686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:11.883686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:11.884142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:11.884688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:11.884688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:11.884688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:11.884688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:11.884688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:11.884688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.884688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:11.904606 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:11.905029 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:11.905250 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:11.905461 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:11.907953 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:11.908180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.908629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:11.908629 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:11.908629 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:11.908629 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:11.908629 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:11.908629 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:11.908629 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.908629 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:11.909300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.909300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:11.909300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:11.909300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:11.909300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.909300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:11.909300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:11.909300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:11.909300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.909300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:11.909300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:11.909300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:11.909300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:11.909300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:11.909300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:11.909762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:11.909762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:11.909762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:11.909762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:11.909762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:11.909762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:11.909762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:11.909762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.909762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:11.910244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.910244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:11.910244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:11.910244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:11.910244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.910244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:11.910244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:11.910244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:11.910244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:11.910244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.910244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:11.910626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:11.910626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:11.910626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.910626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:11.910626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:11.910626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:11.910626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:11.910626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:11.910626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:11.910626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:11.910626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:11.910626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:11.910626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:11.910626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:11.911025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:11.911025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:11.911025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.911025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.911646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:11.912167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:11.912706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:11.912706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:11.912706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.912706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:11.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:11.953128 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:11.953259 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:11.953426 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:11.953763 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:11.953949 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:11.954172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.954779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:11.954779 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:11.954779 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:11.954779 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:11.954779 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:11.954779 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:11.954779 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.954779 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:11.955692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.955692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:11.955692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:11.955692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:11.955692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.955692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:11.955692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:11.955692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.955692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:11.955692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:11.955692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:11.955692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:11.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:11.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:11.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:11.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:11.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:11.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:11.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:11.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:11.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:11.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:11.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:11.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:11.956349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.956349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:11.957027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.957027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:11.957027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:11.957027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:11.957027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:11.957027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:11.957027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.957027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:11.957027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:11.957027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.957027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:11.957027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:11.957027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.957027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:11.957027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:11.957027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:11.957420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:11.957934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:11.957934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:11.957934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:11.964717 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:11.966161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:11.966161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:11.966492 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:12.011494 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:12.011748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:12.012249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:12.012249 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:12.012249 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:12.012249 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:12.012249 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:12.012249 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:12.012249 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:12.012249 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:12.013367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:12.013367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:12.013367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:12.013367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:12.013367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:12.013367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:12.013367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:12.013367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:12.013367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:12.013367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:12.013367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:12.013367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:12.013649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:12.013649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:12.013649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:12.013649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:12.013649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:12.013649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:12.013649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:12.013649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:12.013649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:12.013649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:12.013649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:12.013649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:12.013649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:12.013895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:12.013895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:12.014577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:12.014577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:12.014577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:12.014577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:12.014577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:12.014577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:12.014577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:12.014577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:12.014577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:12.014577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:12.014577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:12.014577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:12.014577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:12.014577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:12.014577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:12.014577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:12.014886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:12.015185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:12.015185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:12.015185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:16.836619 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:16.837360 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:16.837360 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:16.838091 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:16.838284 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:16.838427 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:16.838507 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:16.838627 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:16.838758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:16.838873 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:16.838947 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:16.839080 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:16.839154 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:16.839238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.839778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:16.839778 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:16.839778 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:16.839778 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:16.839778 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:16.839778 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:16.839778 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.839778 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:16.840397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.840397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:16.840397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:16.840397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:16.840397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.840397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:16.840397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:16.840397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.840397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:16.840397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:16.840397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:16.840397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.840797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:16.840797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:16.840797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:16.840797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:16.840797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:16.840797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:16.840797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:16.840797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:16.840797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:16.840797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:16.840797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:16.840797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:16.840797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:16.841021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.841021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:16.841549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.841549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:16.841549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:16.841549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:16.841549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:16.841549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:16.841549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.841549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:16.841549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:16.841549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.841549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:16.841549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:16.841549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.841549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:16.841549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:16.841549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:16.842558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:16.842896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:16.842896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.842896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:16.875595 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:16.875845 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:16.875960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:16.876045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:36:16.876346 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:16.876525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.876984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:16.876984 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:16.876984 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:16.876984 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:16.876984 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:16.876984 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:16.876984 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.876984 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:16.877560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.877560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:16.877560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:16.877560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:16.877560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.877560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:16.877560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:16.877560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:16.877560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.877560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:16.877560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:16.877560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:16.877560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:16.877560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:16.877560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:16.877891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:16.877891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:16.877891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:16.877891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:16.877891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:16.877891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:16.877891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:16.877891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.877891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:16.878267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.878267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:16.878267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:16.878267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:16.878267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.878267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:16.878267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:16.878267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:36:16.878267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:16.878267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:16.878267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.878267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:16.878617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:16.878617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:16.878617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.878617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:16.878617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:16.878617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:16.878617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:16.878617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:16.878617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:16.878617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:16.878617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:16.878617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:16.878617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:16.878617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:16.878879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:16.878879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:16.878879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.878879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:16.878879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.878879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:16.878879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:16.878879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:16.878879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:16.878879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:16.878879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.878879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:16.878879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:16.878879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:16.879124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:16.879419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:16.879419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:16.879419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:16.879419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:16.879419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:16.879419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:16.879419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:16.879419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:16.879419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:16.879419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:16.879419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.879419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:16.905010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:16.905232 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:16.905297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:16.905422 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:16.905727 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:16.905915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.906293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:16.906293 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:16.906293 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:16.906293 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:16.906293 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:16.906293 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:16.906293 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.906293 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:16.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:16.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:16.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:16.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:16.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:16.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:16.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:16.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:16.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:16.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:16.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:16.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:16.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:16.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:16.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:16.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:16.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:16.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:16.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:16.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:16.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:16.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:16.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:16.907581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.908205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:16.908205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:16.908205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:16.908205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:16.908205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.908205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:16.908205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:16.908205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:16.908205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.908205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:16.908205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:16.908205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:16.908205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:16.908978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:16.908978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:16.908978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:16.908978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:16.908978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:16.908978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:16.908978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:16.908978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:16.908978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:16.908978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.908978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:16.908978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:16.909575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.910064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:16.953252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:16.953513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:16.953668 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:16.953869 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:16.954192 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:16.954441 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:16.954813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.955312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:16.955312 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:16.955312 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:16.955312 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:16.955312 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:16.955312 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:16.955312 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.955312 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:16.956271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.956271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:16.956271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:16.956271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:16.956271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.956271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:16.956271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:16.956271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.956271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:16.956271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:16.956271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:16.956271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.956584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:16.956584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:16.956584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:16.956584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:16.956584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:16.956584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:16.956584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:16.956584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:16.956584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:16.956584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:16.956584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:16.956584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:16.956584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:16.956876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.956876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:16.957371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.957371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:16.957371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:16.957371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:16.957371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:16.957371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:16.957371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.957371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:16.957371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:16.957371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.957371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:16.957371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:16.957371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.957371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:16.957371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:16.957371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:16.957652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:16.958014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:16.958014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:16.958014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:16.965743 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:16.966299 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:16.966299 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:16.966661 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:17.022463 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:17.022958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:17.023681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:17.023681 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:17.023681 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:17.023681 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:17.023681 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:17.023681 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:17.023681 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:17.023681 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:17.024391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:17.024391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:17.024391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:17.024391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:17.024391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:17.024391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:17.024391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:17.024391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:17.024391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:17.024391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:17.024391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:17.024391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:17.024750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:17.024750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:17.024750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:17.024750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:17.024750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:17.024750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:17.024750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:17.024750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:17.024750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:17.024750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:17.024750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:17.024750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:17.024750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:17.024975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:17.024975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:17.025652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:17.025652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:17.025652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:17.025652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:17.025652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:17.025652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:17.025652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:17.025652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:17.025652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:17.025652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:17.025652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:17.025652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:17.025652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:17.025652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:17.025652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:17.025652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:17.026530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:17.026913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:17.026913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:17.026913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:21.837712 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:21.838197 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:21.838197 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:21.839312 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:21.839596 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:21.839771 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:21.839893 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:21.840043 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:21.840307 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:21.840307 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:21.840534 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:21.840534 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:21.840534 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:21.840651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.841166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:21.841166 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:21.841166 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:21.841166 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:21.841166 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:21.841166 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:21.841166 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.841166 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:21.842206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.842206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:21.842206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:21.842206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:21.842206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.842206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:21.842206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:21.842206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.842206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:21.842206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:21.842206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:21.842206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.843151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:21.843151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:21.843151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:21.843151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:21.843151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:21.843151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:21.843151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:21.843151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:21.843151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:21.843151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:21.843151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:21.843151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:21.843151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:21.843432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.843432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:21.843432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.843432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:21.843432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:21.843432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:21.843432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:21.843432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:21.843432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.843432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:21.843432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:21.843432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.843432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:21.843432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:21.843432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:21.843799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:21.845375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:21.845375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:21.845375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:21.845375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:21.845375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.845375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:21.876179 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:21.876425 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:21.876567 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:21.876727 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:36:21.877273 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:21.877612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.878552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:21.878552 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:21.878552 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:21.878552 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:21.878552 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:21.878552 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:21.878552 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.878552 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:21.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:21.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:21.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:21.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:21.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:21.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:21.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:21.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:21.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:21.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:21.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:21.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:21.879394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:21.879394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:21.879394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:21.879394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:21.879394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:21.879394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:21.879394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:21.879394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.879394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:21.879828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.879828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:21.879828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:21.879828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:21.879828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.879828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:21.879828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:21.879828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:36:21.879828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:21.879828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:21.879828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.879828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:21.880175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:21.880175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:21.880175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.880175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:21.880175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:21.880175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:21.880175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:21.880175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:21.880175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:21.880175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:21.880175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:21.880175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:21.880175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:21.880175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:21.880489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:21.880489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:21.880489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.880489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:21.880697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.880697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:21.880697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:21.880697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:21.880697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:21.880697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:21.880697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.880697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:21.880697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:21.880697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.880697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:21.880697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:21.880697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.880697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:21.880697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:21.880697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:21.881051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:21.881489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:21.881489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:21.881489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:21.881489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:21.881489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:21.881489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.881489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:21.905685 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:21.905685 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:21.905778 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:21.905875 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:21.906158 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:21.906361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:21.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:21.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:21.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:21.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:21.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:21.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.906852 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:21.907467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.907467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:21.907467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:21.907467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:21.907467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.907467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:21.907467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:21.907467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:21.907467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.907467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:21.907467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:21.907467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:21.907467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:21.907467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:21.907467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:21.907825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:21.907825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:21.907825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:21.907825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:21.907825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:21.907825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:21.907825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:21.907825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.907825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:21.908360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.908360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:21.908360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:21.908360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:21.908360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.908360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:21.908360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:21.908360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:21.908360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:21.908360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.908360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:21.909027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:21.909027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:21.909027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.909027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:21.909027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:21.909027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:21.909027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:21.909027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:21.909027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:21.909027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:21.909027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:21.909027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:21.909027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:21.909027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:21.909504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:21.909504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:21.909504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.909504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:21.909504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.909504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:21.909504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:21.909504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:21.909504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:21.909504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:21.909504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.909504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:21.909504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:21.909504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:21.910033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:21.910692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:21.910692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:21.910692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:21.910692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:21.910692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:21.910692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:21.910692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:21.910692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:21.910692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:21.910692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.910692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:21.953607 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:21.953889 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:21.953999 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:21.954183 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:21.954485 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:21.954686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:21.954943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.955372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:21.955372 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:21.955372 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:21.955372 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:21.955372 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:21.955372 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:21.955372 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.955372 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:21.956106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.956106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:21.956106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:21.956106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:21.956106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.956106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:21.956106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:21.956106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.956106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:21.956106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:21.956106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:21.956106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.956488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:21.956488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:21.956488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:21.956488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:21.956488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:21.956488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:21.956488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:21.956488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:21.956488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:21.956488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:21.956488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:21.956488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:21.956488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:21.956928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.956928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:21.957170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.957170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:21.957170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:21.957170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:21.957170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:21.957170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:21.957170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.957170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:21.957170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:21.957170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.957170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:21.957170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:21.957170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.957170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:21.957170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:21.957170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:21.957772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:21.958257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:21.958257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:21.958257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:21.966479 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:21.966766 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:21.966877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:21.967449 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:21.978203 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=240 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:36:22.001653 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:22.001909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:22.002521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:22.002521 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:22.002521 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:22.002521 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:22.002521 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:22.002521 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:22.002521 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:22.002521 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:22.003510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:22.003510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:22.003510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:22.003510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:22.003510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:22.003510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:22.003510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:22.003510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:22.003510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:22.003510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:22.003510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:22.003510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:22.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:22.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:22.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:22.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:22.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:22.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:22.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:22.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:22.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:22.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:22.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:22.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:22.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:22.004526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:22.004526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:22.004526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:22.004526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:22.004526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:22.004526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:22.004526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:22.004526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:22.004526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:22.004526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:22.004526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:22.004526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:22.004526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:22.004526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:22.004526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:22.005020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:22.005528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:22.005528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:22.005528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:22.005528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:22.005528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:22.005528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:26.838980 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:26.839616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:26.839616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:26.840875 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:26.841037 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:26.841155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:26.841260 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:26.841392 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:26.841493 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:26.841633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:26.841751 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:26.841867 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:26.842044 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:26.842129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.842622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:26.842622 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:26.842622 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:26.842622 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:26.842622 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:26.842622 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:26.842622 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.842622 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:26.843552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.843552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:26.843552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:26.843552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:26.843552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.843552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:26.843552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:26.843552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.843552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:26.843552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:26.843552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:26.843552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.843847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:26.843847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:26.843847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:26.843847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:26.843847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:26.843847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:26.843847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:26.843847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:26.843847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:26.843847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:26.843847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:26.843847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:26.843847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:26.844140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.844140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:26.844630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.844630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:26.844630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:26.844630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:26.844630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:26.844630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:26.844630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.844630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:26.844630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:26.844630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.844630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:26.844630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:26.844630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.844630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:26.844630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:26.844630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:26.844965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:26.845287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:26.845287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.845287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:26.876622 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:26.877111 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:26.877239 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:26.877371 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:36:26.878080 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:26.878303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.879149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:26.879149 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:26.879149 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:26.879149 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:26.879149 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:26.879149 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:26.879149 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.879149 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:26.879926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.879926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:26.879926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:26.879926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:26.879926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.879926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:26.879926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:26.879926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:26.879926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.879926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:26.879926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:26.879926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:26.879926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:26.879926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:26.879926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:26.880375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:26.880375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:26.880375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:26.880375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:26.880375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:26.880375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:26.880375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:26.880375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.880375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:26.880822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.880822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:26.880822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:26.880822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:26.880822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.880822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:26.880822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:26.880822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:36:26.880822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:26.880822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:26.880822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.880822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:26.881194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:26.881194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:26.881194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.881194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:26.881194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:26.881194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:26.881194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:26.881194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:26.881194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:26.881194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:26.881194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:26.881194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:26.881194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:26.881194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:26.881688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:26.881688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:26.881688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.881688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:26.882316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.882316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:26.882316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:26.882316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:26.882316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:26.882316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:26.882316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.882316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:26.882316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:26.882316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.882316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:26.882316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:26.882316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.882316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:26.882316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:26.882316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:26.882831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:26.883450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:26.883450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:26.883450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:26.883450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:26.883450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:26.883450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.883450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:26.905798 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:26.906169 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:26.906289 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:26.906413 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:26.906799 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:26.907000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.907501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:26.907501 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:26.907501 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:26.907501 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:26.907501 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:26.907501 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:26.907501 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.907501 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:26.908156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.908156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:26.908156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:26.908156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:26.908156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.908156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:26.908156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:26.908156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:26.908156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.908156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:26.908156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:26.908156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:26.908156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:26.908156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:26.908156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:26.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:26.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:26.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:26.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:26.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:26.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:26.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:26.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:26.908823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.908823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:26.908823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:26.908823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:26.908823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.908823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:26.908823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:26.908823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:26.908823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:26.908823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.908823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:26.909107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:26.909107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:26.909107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.909107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:26.909107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:26.909107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:26.909107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:26.909107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:26.909107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:26.909107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:26.909107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:26.909107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:26.909107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:26.909107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:26.909348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:26.909348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:26.909348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.909348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.910234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:26.910708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:26.911042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:26.911042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:26.911042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.911042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:26.954349 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:26.954349 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:26.954476 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:26.954686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:26.955168 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:26.955444 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:26.955717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.956177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:26.956177 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:26.956177 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:26.956177 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:26.956177 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:26.956177 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:26.956177 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.956177 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:26.957203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.957203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:26.957203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:26.957203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:26.957203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.957203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:26.957203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:26.957203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.957203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:26.957203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:26.957203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:26.957203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.957709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:26.957709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:26.957709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:26.957709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:26.957709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:26.957709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:26.957709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:26.957709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:26.957709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:26.957709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:26.957709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:26.957709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:26.957709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:26.957916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.957916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:26.958500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.958500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:26.958500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:26.958500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:26.958500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:26.958500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:26.958500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.958500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:26.958500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:26.958500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.958500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:26.958500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:26.958500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.958500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:26.958500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:26.958500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:26.958847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:26.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:26.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:26.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:26.966839 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:26.967182 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:26.967357 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:26.968181 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:27.004780 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:27.005123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:27.005722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:27.005722 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:27.005722 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:27.005722 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:27.005722 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:27.005722 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:27.005722 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:27.005722 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:27.006698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:27.006698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:27.006698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:27.006698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:27.006698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:27.006698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:27.006698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:27.006698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:27.006698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:27.006698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:27.006698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:27.006698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:27.007048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:27.007048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:27.007048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:27.007048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:27.007048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:27.007048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:27.007048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:27.007048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:27.007048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:27.007048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:27.007048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:27.007048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:27.007048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:27.007997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:27.007997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:27.008724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:27.009429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:27.009429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:27.009429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:31.839394 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:31.839947 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:31.839947 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:31.841009 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:31.841260 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:31.841440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:31.841656 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:31.841818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:31.841963 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:31.842110 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:31.842248 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:31.842383 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:31.842512 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:31.842674 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-94baac43-0354-4e19-b8f0-687282fc7eec, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:31.842809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.843388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:31.843388 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:31.843388 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:31.843388 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:31.843388 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:31.843388 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:31.843388 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.843388 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:31.844188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.844188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:31.844188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:31.844188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:31.844188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.844188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:31.844188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:31.844188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.844188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:31.844188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:31.844188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:31.844188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.844518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:31.844518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:31.844518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:31.844518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:31.844518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:31.844518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:31.844518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:31.844518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:31.844518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:31.844518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:31.844518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:31.844518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:31.844518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:31.844820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.844820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:31.844911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.844911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:31.844911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:31.844911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:31.844911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:31.844911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:31.844911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.844911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:31.844911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:31.844911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.844911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:31.844911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:31.844911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.844911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:31.844911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:31.844911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:31.845202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:31.845533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:31.845533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.845533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:31.877488 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:31.877488 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:31.877933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:31.878111 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:36:31.878894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:31.879083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.879567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:31.879567 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:31.879567 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:31.879567 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:31.879567 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:31.879567 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:31.879567 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.879567 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:31.880217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.880217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:31.880217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:31.880217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:31.880217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.880217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:31.880217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:31.880217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:31.880217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.880217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:31.880217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:31.880217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:31.880217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:31.880217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:31.880217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:31.880675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:31.880675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:31.880675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:31.880675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:31.880675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:31.880675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:31.880675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:31.880675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.880675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:31.880921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.880921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:31.880921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:31.880921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:31.880921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.880921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:31.880921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:31.880921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:36:31.880921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:31.880921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:31.880921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.880921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:31.881232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:31.881232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:31.881232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.881232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:31.881232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:31.881232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:31.881232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:31.881232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:31.881232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:31.881232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:31.881232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:31.881232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:31.881232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:31.881232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:31.881568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:31.881568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:31.881568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.881568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:31.882128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.882128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:31.882128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:31.882128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:31.882128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:31.882128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:31.882128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.882128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:31.882128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:31.882128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.882128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:31.882128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:31.882128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.882128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:31.882128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:31.882128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:31.882574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:31.883143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:31.883143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:31.883143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:31.883143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:31.883143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:31.883143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.883143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:31.906276 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:31.906702 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:31.906889 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:31.906962 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:31.907356 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:31.907644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.908188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:31.908188 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:31.908188 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:31.908188 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:31.908188 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:31.908188 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:31.908188 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.908188 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:31.908866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.908866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:31.908866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:31.908866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:31.908866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.908866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:31.908866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:31.908866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:31.908866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.908866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:31.908866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:31.908866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:31.908866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:31.908866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:31.908866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:31.909124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:31.909124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:31.909124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:31.909124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:31.909124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:31.909124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:31.909124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:31.909124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.909124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:31.909641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.909641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:31.909641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:31.909641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:31.909641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.909641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:31.909641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:31.909641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:31.909641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:31.909641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.909641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:31.909919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:31.909919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:31.909919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.909919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:31.909919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:31.909919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:31.909919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:31.909919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:31.909919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:31.909919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:31.909919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:31.909919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:31.909919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:31.909919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:31.910139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:31.910139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:31.910139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.910139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.910715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:31.911011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:31.911296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:31.911296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:31.911296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.911296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:31.955398 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:31.956561 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:31.956762 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:31.957057 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:31.957814 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:31.958100 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:31.958450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.959094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:31.959094 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:31.959094 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:31.959094 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:31.959094 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:31.959094 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:31.959094 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.959094 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:31.960979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.960979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:31.960979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:31.960979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:31.960979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.960979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:31.960979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:31.960979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.960979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:31.960979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:31.960979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:31.960979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.961443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:31.961443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:31.961443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:31.961443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:31.961443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:31.961443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:31.961443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:31.961443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:31.961443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:31.961443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:31.961443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:31.961443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:31.961443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:31.962050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.962050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:31.963652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.963652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:31.963652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:31.963652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:31.963652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:31.963652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:31.963652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.963652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:31.963652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:31.963652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.963652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:31.963652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:31.963652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.963652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:31.963652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:31.963652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:31.964118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:31.964689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:31.964689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:31.964689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:31.968115 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:31.969815 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:31.970030 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:31.971155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:32.035563 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:32.035736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:32.038354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:32.038354 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:32.038354 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:32.038354 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:32.038354 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:32.038354 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:32.038354 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:32.038354 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:32.039729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.068 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:32.039729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:32.039729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:32.039729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:32.039729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:32.039729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:32.039729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:32.039729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:32.039729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:32.039729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:32.039729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:32.039729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:32.040317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:32.040317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:32.040317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:32.040317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:32.040317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:32.040317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:32.040317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:32.040317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:32.040317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:32.040317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:32.040317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:32.040317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:32.040317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:32.040657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:32.040657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:32.040657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:32.040657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:32.040657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:32.040657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:32.040657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:32.040657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:32.040657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:32.040657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:32.040657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:32.040657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:32.040657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:32.040657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:32.040657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:32.041029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:32.041436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:32.041436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:32.041436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:32.041436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:32.041436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:32.041436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:36.839921 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:36.842364 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:36.842364 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:36.843858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:36.844093 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:36.844254 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:36.844406 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:36.844632 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:36.844776 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:36.844937 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:36.846164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:36.846333 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:36.846504 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:36.847172 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-94baac43-0354-4e19-b8f0-687282fc7eec, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:36.847172 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-bf3ba795-d551-4ba8-8dbb-5b02c2dad12f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:36.847172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.847885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:36.847885 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:36.847885 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:36.847885 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:36.847885 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:36.847885 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:36.847885 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.847885 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:36.850697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.850697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:36.850697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:36.850697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:36.850697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.850697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:36.850697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:36.850697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.850697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:36.850697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:36.850697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:36.850697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.851088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:36.851088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:36.851088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:36.851088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:36.851088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:36.851088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:36.851088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:36.851088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:36.851088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:36.851088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:36.851088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:36.851088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:36.851088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:36.851402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.851402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:36.851402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.851402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:36.851402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:36.851402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:36.851402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:36.851402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:36.851402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.851402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:36.851402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:36.851402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.851402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:36.851402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:36.851402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:36.851822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:36.852326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:36.852326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:36.852326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:36.852326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:36.852326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.852326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:36.877636 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:36.878885 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:36.879933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:36.880232 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:36:36.881553 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:36.881553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.881899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:36.881899 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:36.881899 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:36.881899 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:36.881899 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:36.881899 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:36.881899 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.881899 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:36.883570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.883570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:36.883570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:36.883570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:36.883570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.883570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:36.883570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:36.883570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:36.883570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.883570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:36.883570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:36.883570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:36.883570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:36.883570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:36.883570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:36.883991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:36.883991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:36.883991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:36.883991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:36.883991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:36.883991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:36.883991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:36.883991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.883991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:36.884953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.884953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:36.884953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:36.884953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:36.884953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.884953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:36.884953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:36.884953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:36:36.884953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:36.884953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:36.884953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.884953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:36.885318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:36.885318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:36.885318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.885318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:36.885318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:36.885318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:36.885318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:36.885318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:36.885318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:36.885318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:36.885318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:36.885318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:36.885318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:36.885318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:36.885663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:36.885663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:36.885663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.885663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:36.885797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.885797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:36.885797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:36.885797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:36.885797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:36.885797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:36.885797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.885797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:36.885797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:36.885797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.885797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:36.885797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:36.885797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.885797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:36.885797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:36.885797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:36.886174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:36.886579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:36.886579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:36.886579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:36.886579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:36.886579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:36.886579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.886579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:36.908021 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:36.908998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:36.909138 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:36.909253 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:36.910942 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:36.911216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.912644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:36.912644 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:36.912644 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:36.912644 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:36.912644 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:36.912644 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:36.912644 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.912644 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:36.914613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.914613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:36.914613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:36.914613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:36.914613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.914613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:36.914613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:36.914613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:36.914613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.914613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:36.914613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:36.914613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:36.914613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:36.914613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:36.914613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:36.915135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:36.915135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:36.915135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:36.915135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:36.915135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:36.915135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:36.915135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:36.915135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.915135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:36.917197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.917197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:36.917197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:36.917197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:36.917197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.917197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:36.917197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:36.917197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:36.917197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:36.917197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.917197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:36.917426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:36.917426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:36.917426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.917426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:36.917426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:36.917426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:36.917426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:36.917426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:36.917426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:36.917426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:36.917426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:36.917426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:36.917426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:36.917426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:36.917716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:36.917716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:36.917716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.917716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.918383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:36.918725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:36.919006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:36.919006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:36.919006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.919006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:36.956563 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:36.956954 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:36.957075 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:36.957260 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:36.957735 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:36.957992 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:36.958285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.958701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:36.958701 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:36.958701 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:36.958701 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:36.958701 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:36.958701 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:36.958701 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.958701 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:36.959721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.959721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:36.959721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:36.959721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:36.959721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.959721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:36.959721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:36.959721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.959721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:36.959721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:36.959721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:36.959721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.959939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:36.959939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:36.959939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:36.959939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:36.959939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:36.959939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:36.959939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:36.959939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:36.959939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:36.959939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:36.959939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:36.959939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:36.959939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:36.960182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.960182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:36.961028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.961028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:36.961028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:36.961028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:36.961028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:36.961028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:36.961028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.961028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:36.961028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:36.961028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.961028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:36.961028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:36.961028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.961028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:36.961028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:36.961028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:36.961324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:36.961689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:36.961689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:36.961689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:36.970561 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:36.970984 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:36.971105 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:36.971891 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:37.025284 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:37.025531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:37.026075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:37.026075 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:37.026075 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:37.026075 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:37.026075 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:37.026075 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:37.026075 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:37.026075 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:37.027145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:37.027145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:37.027145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:37.027145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:37.027145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:37.027145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:37.027145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:37.027145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:37.027145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:37.027145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:37.027145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:37.027145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:37.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:37.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:37.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:37.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:37.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:37.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:37.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:37.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:37.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:37.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:37.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:37.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:37.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:37.027967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:37.027967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:37.028465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:37.028465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:37.028465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:37.028465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:37.028465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:37.028465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:37.028465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:37.028465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:37.028465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:37.028465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:37.028465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:37.028465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:37.028465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:37.028465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:37.028465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:37.028465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:37.028929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:37.029397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:37.029397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:37.029397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:41.841222 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:41.841868 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:41.841868 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:41.842753 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:41.842932 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:41.843055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:41.843181 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:41.843323 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:41.843459 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:41.843645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:41.843769 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:41.843886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:41.843998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:41.844114 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-94baac43-0354-4e19-b8f0-687282fc7eec, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:41.844220 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-bf3ba795-d551-4ba8-8dbb-5b02c2dad12f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:41.844336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.844864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:41.844864 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:41.844864 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:41.844864 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:41.844864 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:41.844864 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:41.844864 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.844864 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:41.845672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.845672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:41.845672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:41.845672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:41.845672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.845672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:41.845672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:41.845672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.845672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:41.845672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:41.845672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:41.845672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.846033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:41.846033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:41.846033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:41.846033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:41.846033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:41.846033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:41.846033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:41.846033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:41.846033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:41.846033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:41.846033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:41.846033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:41.846033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:41.846414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.846414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:41.846986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.846986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:41.846986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:41.846986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:41.846986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:41.846986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:41.846986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.846986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:41.846986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:41.846986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.846986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:41.846986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:41.846986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.846986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:41.846986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:41.846986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:41.847648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:41.848441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:41.848441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.848441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:41.879115 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:41.880342 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:41.880534 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:41.880688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:36:41.881099 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:41.881307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.883878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:41.883878 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:41.883878 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:41.883878 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:41.883878 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:41.883878 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:41.883878 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.883878 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:41.884741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.884741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:41.884741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:41.884741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:41.884741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.884741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:41.884741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:41.884741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:41.884741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.884741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:41.884741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:41.884741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:41.884741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:41.884741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:41.884741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:41.885053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:41.885053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:41.885053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:41.885053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:41.885053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:41.885053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:41.885053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:41.885053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.885053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:41.887576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.887576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:41.887576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:41.887576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:41.887576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.887576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:41.887576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:41.887576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:36:41.887576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:41.887576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:41.887576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.887576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:41.887830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:41.887830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:41.887830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.887830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:41.887830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:41.887830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:41.887830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:41.887830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:41.887830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:41.887830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:41.887830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:41.887830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:41.887830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:41.887830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:41.888075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:41.888075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:41.888075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.888075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:41.890996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.890996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:41.890996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:41.890996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:41.890996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:41.890996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:41.890996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.890996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:41.890996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:41.890996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.890996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:41.890996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:41.890996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.890996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:41.890996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:41.890996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:41.891320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:41.891695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:41.891695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:41.891695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:41.891695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:41.891695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:41.891695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.891695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:41.912251 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:41.915596 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:41.915596 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:41.915596 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:41.919010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:41.919010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.919010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:41.919010 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:41.919010 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:41.919010 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:41.919010 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:41.919010 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:41.919010 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.919010 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:41.919010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.919010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:41.919010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:41.919010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:41.919625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:41.921589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.921589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:41.921589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.921589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:41.921589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:41.921589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:41.921589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.921589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:41.921589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:41.921589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:41.921589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:41.921589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.921830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:41.921830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:41.921830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:41.921830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.921830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:41.921830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:41.921830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:41.921830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:41.921830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:41.921830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:41.921830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:41.921830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:41.921830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:41.921830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:41.922053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:41.922053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:41.922053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:41.922053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.922053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:41.922053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.922053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:41.922053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:41.922053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:41.922053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:41.922053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:41.922053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.922053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:41.922053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:41.922053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:41.922292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:41.922626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:41.922626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:41.922626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:41.922626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:41.922626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:41.922626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:41.922626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:41.922626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:41.922626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:41.922626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.922626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:41.957019 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:41.957524 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:41.957689 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:41.957857 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:41.958849 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:41.959169 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:41.962132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.962761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:41.962761 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:41.962761 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:41.962761 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:41.962761 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:41.962761 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:41.962761 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.962761 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:41.963324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.963324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:41.963324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:41.963324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:41.963324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.963324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:41.963324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:41.963324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.963324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:41.963324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:41.963324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:41.963324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.963705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:41.963705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:41.963705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:41.963705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:41.963705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:41.963705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:41.963705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:41.963705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:41.963705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:41.963705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:41.963705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:41.963705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:41.963705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:41.964047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.964047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:41.964111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.964111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:41.964111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:41.964111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:41.964111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:41.964111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:41.964111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.964111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:41.964111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:41.964111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.964111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:41.964111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:41.964111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.964111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:41.964111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:41.964111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:41.964493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:41.964991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:41.964991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:41.964991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:41.971056 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:41.971523 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:41.971646 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:41.972252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:42.017085 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:42.018691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:42.019526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:42.019526 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:42.019526 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:42.019526 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:42.019526 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:42.019526 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:42.019526 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:42.019526 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:42.020673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:42.020673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:42.020673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:42.020673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:42.020673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:42.020673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:42.020673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:42.020673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:42.020673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:42.020673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:42.020673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:42.020673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:42.021125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:42.021125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:42.021125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:42.021125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:42.021125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:42.021125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:42.021125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:42.021125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:42.021125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:42.021125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:42.021125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:42.021125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:42.021125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:42.021384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:42.021384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:42.021720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:42.021720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:42.021720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:42.021720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:42.021720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:42.021720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:42.021720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:42.021720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:42.021720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:42.021720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:42.021720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:42.021720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:42.021720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:42.021720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:42.021720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:42.021720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:42.022118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:42.022422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:42.022422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:42.022422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:46.841695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:46.842651 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:46.842805 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:46.844051 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:46.844291 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:46.844450 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:46.844736 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:46.844975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:46.845129 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:46.845278 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:46.845411 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:46.845593 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:46.845734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-94baac43-0354-4e19-b8f0-687282fc7eec, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:46.845911 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-bf3ba795-d551-4ba8-8dbb-5b02c2dad12f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:46.846051 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:46.846188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.847309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:46.847309 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:46.847309 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:46.847309 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:46.847309 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:46.847309 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:46.847309 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.847309 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:46.848045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.848045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:46.848045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:46.848045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:46.848045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.848045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:46.848045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:46.848045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.848045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:46.848045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:46.848045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:46.848045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.848446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:46.848446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:46.848446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:46.848446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:46.848446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:46.848446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:46.848446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:46.848446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:46.848446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:46.848446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:46.848446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:46.848446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:46.848446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:46.848891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.848891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:46.848891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.848891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:46.848891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:46.848891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:46.848891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:46.848891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:46.848891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.848891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:46.848891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:46.848891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.848891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:46.848891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:46.848891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:46.849402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:46.849938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:46.849938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:46.849938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:46.849938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:46.849938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.849938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:46.880335 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:46.881187 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:46.881359 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:46.881523 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:36:46.882009 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:46.882216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.882764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:46.882764 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:46.882764 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:46.882764 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:46.882764 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:46.882764 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:46.882764 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.882764 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:46.884117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.884117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:46.884117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:46.884117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:46.884117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.884117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:46.884117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:46.884117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:46.884117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.884117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:46.884117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:46.884117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:46.884117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:46.884117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:46.884117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:46.884717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:46.884717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:46.884717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:46.884717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:46.884717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:46.884717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:46.884717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:46.884717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.884717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:46.885124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.885124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:46.885124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:46.885124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:46.885124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.885124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:46.885124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:46.885124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:36:46.885124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:46.885124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:46.885124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.885124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:46.885609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:46.885609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:46.885609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.885609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:46.885609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:46.885609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:46.885609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:46.885609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:46.885609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:46.885609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:46.885609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:46.885609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:46.885609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:46.885609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:46.885980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:46.885980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:46.885980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.885980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:46.886814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.886814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:46.886814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:46.886814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:46.886814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:46.886814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:46.886814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.886814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:46.886814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:46.886814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.886814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:46.886814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:46.886814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.886814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:46.886814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:46.886814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:46.887276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:46.887978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:46.887978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:46.887978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:46.887978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:46.887978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:46.887978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.887978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:46.914527 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:46.914772 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:46.916437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:46.916437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:46.916437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:46.916437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.916437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:46.916437 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:46.916437 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:46.916437 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:46.916437 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:46.916437 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:46.916437 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.916437 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:46.917240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.917240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:46.917240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:46.917240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:46.917240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.917240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:46.917240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:46.917240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:46.917240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.917240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:46.917240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:46.917240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:46.917240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:46.917240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:46.917240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:46.917498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:46.917498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:46.917498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:46.917498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:46.917498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:46.917498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:46.917498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:46.917498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.917498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:46.917704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.917704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:46.917704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:46.917704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:46.917704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.917704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:46.917704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:46.917704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:46.917704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:46.917704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.917704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:46.917880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:46.917880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:46.917880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.917880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:46.917880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:46.917880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:46.917880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:46.917880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:46.917880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:46.917880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:46.917880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:46.917880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:46.917880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:46.917880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:46.918127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:46.918127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:46.918127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.918127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.918308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:46.918718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:46.919028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:46.919028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:46.919028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.919028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:46.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:46.958651 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:46.958804 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:46.959016 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:46.962498 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:46.962498 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:46.962498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.962498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:46.962498 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:46.962498 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:46.962498 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:46.962498 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:46.962498 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:46.962498 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.962498 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:46.963039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.963039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:46.963039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:46.963039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:46.963039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.963039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:46.963039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:46.963039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.963039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:46.963039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:46.963039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:46.963039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.963381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:46.963381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:46.963381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:46.963381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:46.963381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:46.963381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:46.963381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:46.963381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:46.963381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:46.963381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:46.963381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:46.963381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:46.963381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:46.963827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.963827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:46.963930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.963930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:46.963930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:46.963930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:46.963930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:46.963930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:46.963930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.963930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:46.963930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:46.963930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.963930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:46.963930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:46.963930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.963930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:46.963930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:46.963930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:46.964280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:46.964662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:46.964662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:46.964662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:46.971716 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:46.972160 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:46.972298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:46.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:47.011099 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:47.011339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:47.011944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:47.011944 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:47.011944 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:47.011944 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:47.011944 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:47.011944 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:47.011944 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:47.011944 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:47.012779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:47.012779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:47.012779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:47.012779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:47.012779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:47.012779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:47.012779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:47.012779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:47.012779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:47.012779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:47.012779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:47.012779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:47.013104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:47.013104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:47.013104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:47.013104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:47.013104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:47.013104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:47.013104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:47.013104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:47.013104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:47.013104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:47.013104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:47.013104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:47.013104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:47.013358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:47.013358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:47.013701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:47.013701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:47.013701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:47.013701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:47.013701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:47.013701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:47.013701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:47.013701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:47.013701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:47.013701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:47.013701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:47.013701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:47.013701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:47.013701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:47.013701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:47.013701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:47.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:47.014307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:47.014307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:47.014307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:51.843857 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:51.844807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:51.844807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:51.846529 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:51.846975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:51.847324 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:51.847717 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:51.848034 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:51.848348 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:51.848649 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:51.848930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:51.849184 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:51.850107 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:51.850107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.850387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:51.850387 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:51.850387 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:51.850387 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:51.850387 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:51.850387 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:51.850387 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.850387 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:51.851259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.851259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:51.851259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:51.851259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:51.851259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.851259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:51.851259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:51.851259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.851259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:51.851259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:51.851259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:51.851259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.851796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:51.851796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:51.851796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:51.851796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:51.851796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:51.851796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:51.851796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:51.851796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:51.851796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:51.851796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:51.851796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:51.851796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:51.851796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:51.852129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.852129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:51.852129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.852129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:51.852129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:51.852129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:51.852129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:51.852129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:51.852129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.852129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:51.852129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:51.852129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.852129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:51.852129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:51.852129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:51.852699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:51.853245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:51.853245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:51.853245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:51.853245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:51.853245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.853245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:51.881584 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:51.881584 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:51.882157 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:51.882242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:36:51.883496 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:51.883496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.883633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:51.883633 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:51.883633 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:51.883633 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:51.883633 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:51.883633 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:51.883633 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.883633 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:51.884375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.884375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:51.884375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:51.884375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:51.884375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.884375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:51.884375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:51.884375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:51.884375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.884375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:51.884375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:51.884375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:51.884375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:51.884375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:51.884375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:51.884905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:51.884905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:51.884905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:51.884905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:51.884905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:51.884905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:51.884905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:51.884905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.884905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:51.885191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.885191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:51.885191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:51.885191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:51.885191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.885191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:51.885191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:51.885191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:36:51.885191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:51.885191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:51.885191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.885191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:51.885584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:51.885584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:51.885584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.885584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:51.885584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:51.885584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:51.885584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:51.885584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:51.885584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:51.885584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:51.885584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:51.885584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:51.885584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:51.885584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:51.885957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:51.885957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:51.885957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.885957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:51.885957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.885957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:51.885957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:51.885957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:51.885957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:51.885957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:51.885957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.885957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:51.885957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:51.885957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:51.886681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:51.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:51.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:51.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:51.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:51.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:51.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:51.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:51.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:51.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:51.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:51.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.887235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:51.915906 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:51.916372 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:51.916719 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:51.916719 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:51.917948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:51.917948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.918161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:51.918161 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:51.918161 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:51.918161 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:51.918161 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:51.918161 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:51.918161 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.918161 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:51.918945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.918945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:51.918945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:51.918945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:51.918945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.918945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:51.918945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:51.918945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:51.918945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.918945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:51.918945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:51.918945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:51.918945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:51.918945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:51.918945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:51.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:51.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:51.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:51.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:51.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:51.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:51.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:51.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:51.919684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.919684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:51.919684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:51.919684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:51.919684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.919684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:51.919684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:51.919684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:51.919684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:51.919684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.919684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:51.919952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:51.919952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:51.919952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.919952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:51.919952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:51.919952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:51.919952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:51.919952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:51.919952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:51.919952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:51.919952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:51.919952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:51.919952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:51.919952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:51.920165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:51.920165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:51.920165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.920165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.920552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:51.920857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:51.921184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:51.921184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:51.921184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.921184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:51.958734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:51.959167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:51.959298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:51.959516 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:51.960998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:51.960998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:51.960998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.961162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:51.961162 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:51.961162 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:51.961162 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:51.961162 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:51.961162 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:51.961162 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.961162 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:51.962646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.962646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:51.962646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:51.962646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:51.962646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.962646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:51.962646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:51.962646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.962646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:51.962646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:51.962646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:51.962646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.963014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:51.963014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:51.963014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:51.963014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:51.963014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:51.963014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:51.963014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:51.963014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:51.963014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:51.963014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:51.963014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:51.963014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:51.963014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:51.963358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.963358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:51.964208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.964208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:51.964208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:51.964208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:51.964208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:51.964208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:51.964208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.964208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:51.964208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:51.964208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.964208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:51.964208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:51.964208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.964208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:51.964208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:51.964208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:51.964773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:51.965451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:51.965451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:51.965451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:51.972319 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:51.972969 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:51.973168 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:51.975587 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:52.047419 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:52.047419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:52.047669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:52.047669 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:52.047669 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:52.047669 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:52.047669 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:52.047669 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:52.047669 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:52.047669 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:52.048874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.074 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:52.048874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:52.048874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:52.048874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:52.048874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:52.048874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:52.048874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:52.048874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:52.048874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:52.048874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:52.048874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:52.048874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:52.049493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:52.049493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:52.049493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:52.049493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:52.049493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:52.049493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:52.049493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:52.049493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:52.049493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:52.049493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:52.049493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:52.049493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:52.049493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:52.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:52.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:52.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:52.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:52.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:52.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:52.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:52.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:52.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:52.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:52.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:52.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:52.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:52.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:52.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:52.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:52.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:52.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:52.050674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:52.051154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:52.051154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:52.051154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:56.844312 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:56.845031 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:56.845031 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:56.845916 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:56.846622 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:56.846984 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:56.847314 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:56.848093 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:56.848456 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:56.848894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:56.849010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:56.849237 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:56.849481 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:56.849749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:56.849986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.852885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:56.852885 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:56.852885 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:56.852885 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:56.852885 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:56.852885 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:56.852885 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.852885 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:56.853857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.853857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:56.853857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:56.853857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:56.853857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.853857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:56.853857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:56.853857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.853857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:56.853857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:56.853857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:56.853857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.856239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:56.856239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:56.856239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:56.856239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:56.856239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:56.856239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:56.856239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:56.856239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:56.856239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:56.856239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:56.856239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:56.856239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:56.856239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:56.856780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.856780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:56.856780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.856780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:56.856780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:56.856780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:56.856780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:56.856780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:56.856780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.856780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:56.856780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:56.856780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.856780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:56.856780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:56.856780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:56.857223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:56.857824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:56.857824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:56.857824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:56.857824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:56.857824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.857824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:56.881719 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:56.882116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:56.882235 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:56.882336 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:36:56.884675 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:56.885003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.886785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:56.886785 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:56.886785 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:56.886785 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:56.886785 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:56.886785 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:56.886785 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.886785 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:56.887555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.887555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:56.887555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:56.887555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:56.887555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.887555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:56.887555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:56.887555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:56.887555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.887555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:56.887555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:56.887555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:56.887555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:56.887555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:56.887555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:56.888136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:56.888136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:56.888136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:56.888136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:56.888136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:56.888136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:56.888136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:56.888136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.888136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:56.888682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.888682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:56.888682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:56.888682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:56.888682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.888682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:56.888682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:56.888682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:36:56.888682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:56.888682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:56.888682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.888682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:56.889037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:56.889037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:56.889037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.889037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:56.889037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:56.889037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:56.889037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:56.889037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:56.889037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:56.889037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:56.889037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:56.889037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:56.889037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:56.889037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:56.889436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:56.889436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:56.889436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.889436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:56.889436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.889436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:56.889436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:56.889436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:56.889436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:56.889436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:56.889436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.889436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:56.889436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:56.889436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:56.890892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:56.891234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:56.891234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:56.891234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:56.891234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:56.891234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:56.891234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:56.891234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:56.891234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:56.891234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:56.891234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:56.891234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.891234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:56.916472 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:56.917478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:56.918152 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:56.918152 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:56.918297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:56.918500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.920627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:56.920627 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:56.920627 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:56.920627 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:56.920627 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:56.920627 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:56.920627 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.920627 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:56.921709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.921709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:36:56.921709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:56.921709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:56.921709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.921709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:56.921709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:56.921709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:36:56.921709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.921709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:36:56.921709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:56.921709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:36:56.921709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:56.921709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:36:56.921709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:36:56.922147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:56.922147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:36:56.922147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:56.922147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:36:56.922147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:36:56.922147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:56.922147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:36:56.922147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.922147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:36:56.923739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.923739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:56.923739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:56.923739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:56.923739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.923739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:56.923739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:56.923739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:56.923739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:56.923739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.923739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:56.924835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:56.924835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:56.924835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.924835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:56.924835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:56.924835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:56.924835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:56.924835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:56.924835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:56.924835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:56.924835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:56.924835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:56.924835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:56.924835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:56.925253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:56.925253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:56.925253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.925253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.926356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:56.926829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:56.927244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:56.927244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:56.927244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.927244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:56.960792 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:56.961430 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:56.961548 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:56.961708 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:56.962232 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:56.962500 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:56.962899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.963480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:56.963480 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:56.963480 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:56.963480 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:56.963480 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:56.963480 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:56.963480 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.963480 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:56.964606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.964606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:56.964606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:56.964606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:56.964606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.964606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:56.964606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:56.964606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.964606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:56.964606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:56.964606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:56.964606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.965000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:56.965000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:56.965000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:56.965000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:56.965000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:56.965000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:56.965000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:56.965000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:56.965000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:56.965000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:56.965000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:56.965000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:56.965000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:56.965335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.965335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:56.966285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.966285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:56.966285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:56.966285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:56.966285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:56.966285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:56.966285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.966285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:56.966285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:56.966285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.966285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:56.966285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:56.966285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.966285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:56.966285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:56.966285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:56.966744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:56.967237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:56.967237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:56.967237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:36:56.974956 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:36:56.975135 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:36:56.975241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:36:56.976310 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:36:57.035926 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:36:57.036112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:57.038294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:36:57.038294 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:57.038294 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:36:57.038294 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:36:57.038294 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:57.038294 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:36:57.038294 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:57.038294 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:36:57.038294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.062 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:57.038294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:36:57.038294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:57.038294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:36:57.038294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:57.038294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:57.038294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:57.038853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:57.038853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:57.038853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:57.038853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:36:57.038853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:57.038853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:36:57.038853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:57.038853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:36:57.038853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:57.038853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:36:57.038853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:36:57.038853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:57.038853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:36:57.038853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:57.039284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:36:57.039284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:36:57.039284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:57.039284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:36:57.039284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:57.039284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:36:57.039284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:57.039284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:36:57.039284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:36:57.039284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:36:57.039284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:36:57.039284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:36:57.039284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:57.039284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:36:57.039769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:36:57.040402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:36:57.040402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:36:57.040402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:36:57.040402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:36:57.040402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:36:57.040402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:36:57.040402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:36:57.040402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:36:57.040402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:01.844815 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:01.845576 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:01.845576 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:01.846834 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:01.848324 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:01.848324 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:01.848324 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:01.848324 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:01.848324 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:01.848655 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:01.848901 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:01.849415 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:01.849820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:01.850206 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:01.850582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.851413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:01.851413 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:01.851413 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:01.851413 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:01.851413 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:01.851413 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:01.851413 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.851413 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:01.852964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.852964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:01.852964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:01.852964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:01.852964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.852964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:01.852964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:01.852964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.852964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:01.852964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:01.852964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:01.852964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.853777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:01.853777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:01.853777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:01.853777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:01.853777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:01.853777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:01.853777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:01.853777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:01.853777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:01.853777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:01.853777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:01.853777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:01.853777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:01.854160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.854160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:01.854160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.854160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:01.854160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:01.854160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:01.854160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:01.854160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:01.854160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.854160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:01.854160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:01.854160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.854160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:01.854160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:01.854160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:01.854742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:01.856904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:01.856904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:01.856904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:01.856904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:01.856904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.856904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:01.882207 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:01.884977 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:01.885100 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:01.885215 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:37:01.885825 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:01.886754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.886846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:01.886846 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:01.886846 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:01.886846 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:01.886846 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:01.886846 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:01.886846 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.886846 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:01.887464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.887464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:01.887464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:01.887464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:01.887464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.887464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:01.887464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:01.887464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:01.887464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.887464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:01.887464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:01.887464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:01.887464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:01.887464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:01.887464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:01.888326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:01.888326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:01.888326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:01.888326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:01.888326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:01.888326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:01.888326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:01.888326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.888326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:01.888326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.888326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:01.888326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:01.888326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:01.888326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.889200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:01.889200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:01.889200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:37:01.889200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:01.889200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:01.889200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.889200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:01.889200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:01.889200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:01.889200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.889200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:01.889200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:01.889200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:01.890177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:01.890177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:01.890177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:01.890177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:01.890177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:01.890177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:01.890177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:01.890177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:01.890177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:01.890177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:01.890177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.890177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:01.890529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.890529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:01.890529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:01.890529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:01.890529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:01.890529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:01.890529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.890529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:01.890529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:01.890529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.890529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:01.890529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:01.890529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.890529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:01.890529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:01.890529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:01.891379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:01.892395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:01.892395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:01.892395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:01.892395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:01.892395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:01.892395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.892395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:01.917603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:01.917936 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:01.918054 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:01.918192 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:01.919139 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:01.919139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.919223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:01.919223 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:01.919223 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:01.919223 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:01.919223 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:01.919223 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:01.919223 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.919223 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:01.919957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.919957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:01.919957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:01.919957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:01.919957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.919957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:01.919957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:01.919957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:01.919957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.919957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:01.919957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:01.919957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:01.919957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:01.919957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:01.919957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:01.920747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:01.920747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:01.920747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:01.920747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:01.920747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:01.920747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:01.920747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:01.920747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.920747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:01.921100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.921100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:01.921100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:01.921100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:01.921100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.921100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:01.921100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:01.921100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:01.921100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:01.921100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.921100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:01.921385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:01.921385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:01.921385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.921385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:01.921385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:01.921385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:01.921385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:01.921385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:01.921385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:01.921385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:01.921385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:01.921385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:01.921385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:01.921385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:01.921666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:01.921666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:01.921666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.921666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.921933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:01.922372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:01.923181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:01.923181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:01.923181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.923181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:01.961292 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:01.963929 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:01.964081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:01.964311 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:01.965914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:01.965914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:01.965914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.966158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:01.966158 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:01.966158 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:01.966158 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:01.966158 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:01.966158 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:01.966158 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.966158 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:01.967398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.967398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:01.967398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:01.967398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:01.967398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.967398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:01.967398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:01.967398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.967398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:01.967398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:01.967398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:01.967398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.967847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:01.967847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:01.967847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:01.967847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:01.967847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:01.967847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:01.967847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:01.967847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:01.967847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:01.967847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:01.967847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:01.967847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:01.967847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:01.968619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.968619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:01.968897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.968897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:01.968897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:01.968897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:01.968897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:01.968897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:01.968897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.968897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:01.968897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:01.968897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.968897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:01.968897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:01.968897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.968897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:01.968897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:01.968897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:01.969359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:01.969909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:01.969909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:01.969909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:01.975290 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:01.976077 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:01.976238 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:01.976942 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:02.022778 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:02.022778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:02.023996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:02.023996 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:02.023996 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:02.023996 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:02.023996 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:02.023996 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:02.023996 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:02.023996 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:02.024866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:02.024866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:02.024866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:02.024866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:02.024866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:02.024866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:02.024866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:02.024866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:02.024866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:02.024866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:02.024866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:02.024866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:02.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:02.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:02.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:02.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:02.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:02.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:02.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:02.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:02.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:02.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:02.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:02.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:02.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:02.026073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:02.026073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:02.026073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:02.026073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:02.026073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:02.026073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:02.026073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:02.026073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:02.026073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:02.026073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:02.026073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:02.026073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:02.026073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:02.026073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:02.026073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:02.026518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:02.026954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:02.026954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:02.026954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:02.026954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:02.026954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:02.026954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:06.845281 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:06.845678 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:06.845768 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:06.847370 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:06.847722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:06.848096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:06.848261 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:06.848394 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:06.848518 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:06.848709 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:06.848842 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:06.848944 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:06.849076 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:06.849179 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:06.849276 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:06.849497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.851337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:06.851337 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:06.851337 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:06.851337 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:06.851337 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:06.851337 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:06.851337 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.851337 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:06.851337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.851337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:06.851337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:06.851337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:06.851337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.851337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:06.851337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:06.851859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.851859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:06.851859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:06.851859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:06.851859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.851859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:06.851859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:06.851859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:06.851859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:06.851859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:06.851859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:06.851859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:06.851859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:06.851859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:06.852236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:06.852236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:06.852236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:06.852236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:06.852236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.852236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:06.852236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.852236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:06.852236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:06.852236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:06.852236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:06.852236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:06.852236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.852236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:06.852636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:06.853138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:06.853138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:06.853138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:06.853138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:06.853138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:06.853138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:06.853138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:06.853138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:06.853138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.853138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:06.885013 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:06.886125 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:06.886297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:06.886451 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:37:06.886975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:06.887248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.888692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:06.888692 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:06.888692 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:06.888692 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:06.888692 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:06.888692 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:06.888692 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.888692 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:06.888989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.888989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:06.888989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:06.888989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:06.888989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.888989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:06.888989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:06.888989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:06.888989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.888989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:06.888989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:06.888989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:06.888989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:06.888989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:06.888989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:06.889363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:06.889363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:06.889363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:06.889363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:06.889363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:06.889363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:06.889363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:06.889363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.889363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:06.889900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.889900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:06.889900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:06.889900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:06.889900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.889900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:06.889900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:06.889900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:37:06.889900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:06.889900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:06.889900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.889900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:06.890227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:06.890227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:06.890227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.890227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:06.890227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:06.890227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:06.890227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:06.890227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:06.890227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:06.890227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:06.890227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:06.890227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:06.890227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:06.890227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:06.892376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:06.892376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:06.892376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.892376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:06.892376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.892376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:06.892376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:06.892376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:06.892376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:06.892376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:06.892376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.892376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:06.892376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:06.892376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:06.892911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:06.893487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:06.893487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:06.893487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:06.893487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:06.893487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:06.893487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:06.893487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:06.893487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:06.893487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:06.893487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:06.893487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.893487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:06.918614 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:06.918968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:06.919087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:06.919216 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:06.919710 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:06.919986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.921730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:06.922357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:06.922357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:06.922357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:06.922357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:06.922357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:06.922357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:06.922357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:06.922357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:06.922357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:06.922357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:06.922357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:06.922357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:06.922357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.922357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:06.923160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.923160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:06.923160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:06.923160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:06.923160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.923160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:06.923160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:06.923160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:06.923160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:06.923160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.923160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:06.923559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:06.923559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:06.923559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.923559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:06.923559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:06.923559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:06.923559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:06.923559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:06.923559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:06.923559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:06.923559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:06.923559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:06.923559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:06.923559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:06.924031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:06.924031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:06.924031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.924031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.925441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:06.925967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:06.926494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:06.926494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:06.926494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.926494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:06.964064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:06.964817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:06.965795 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:06.966600 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:06.966682 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:06.968606 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:06.968606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.968606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:06.968606 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:06.968606 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:06.968606 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:06.968606 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:06.968606 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:06.968606 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.968606 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:06.969462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.969462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:06.969462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:06.969462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:06.969462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.969462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:06.969462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:06.969462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.969462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:06.969462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:06.969462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:06.969462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.969882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:06.969882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:06.969882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:06.969882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:06.969882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:06.969882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:06.969882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:06.969882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:06.969882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:06.969882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:06.969882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:06.969882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:06.969882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:06.970393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.970393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:06.970526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.970526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:06.970526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:06.970526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:06.970526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:06.970526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:06.970526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.970526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:06.970526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:06.970526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.970526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:06.970526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:06.970526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.970526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:06.970526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:06.970526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:06.971983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:06.972365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:06.972365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:06.972365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:06.977083 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:06.977899 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:06.978189 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:06.981752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:07.045300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:07.046883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:07.048274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:07.048274 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:07.048274 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:07.048274 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:07.048274 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:07.048274 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:07.048274 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:07.048274 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:07.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.070 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:07.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:07.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:07.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:07.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:07.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:07.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:07.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:07.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:07.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:07.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:07.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:07.048991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:07.048991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:07.048991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:07.048991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:07.048991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:07.048991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:07.048991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:07.048991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:07.048991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:07.048991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:07.048991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:07.048991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:07.048991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:07.049197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:07.049197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:07.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:07.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:07.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:07.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:07.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:07.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:07.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:07.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:07.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:07.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:07.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:07.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:07.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:07.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:07.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:07.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:07.050339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:07.050925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:07.050925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:07.050925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:11.845789 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:11.846252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:11.846252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:11.849049 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:11.849049 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:11.849049 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:11.849049 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:11.849049 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:11.849242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:11.849242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:11.849242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:11.849242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:11.849242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:11.849349 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:11.849349 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-66741bf2-bf6e-4ea6-9962-92314fbb8568, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:11.849349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.849349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:11.849349 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:11.849349 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:11.849349 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:11.849349 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:11.849349 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:11.849349 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.849349 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:11.849786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.849786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:11.849786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:11.849786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:11.849786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.849786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:11.849786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:11.849786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.849786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:11.849786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:11.849786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:11.849786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.850082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:11.850082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:11.850082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:11.850082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:11.850082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:11.850082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:11.850082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:11.850082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:11.850082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:11.850082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:11.850082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:11.850082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:11.850082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:11.850299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.850299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:11.850418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.850418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:11.850418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:11.850418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:11.850418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:11.850418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:11.850418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.850418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:11.850418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:11.850418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.850418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:11.850418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:11.850418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.850418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:11.850418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:11.850418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:11.850809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:11.851143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:11.851143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.851143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:11.886152 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:11.886911 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:11.886911 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:11.886911 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:37:11.887158 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:11.887314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.888579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:11.888579 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:11.888579 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:11.888579 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:11.888579 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:11.888579 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:11.888579 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.888579 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:11.889104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.889104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:11.889104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:11.889104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:11.889104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.889104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:11.889104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:11.889104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:11.889104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.889104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:11.889104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:11.889104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:11.889104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:11.889104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:11.889104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:11.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:11.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:11.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:11.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:11.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:11.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:11.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:11.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:11.892690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.892690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:11.892690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:11.892690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:11.892690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.892690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:11.892690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:11.892690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:37:11.892690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:11.892690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:11.892690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.892690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:11.893078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:11.893078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:11.893078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.893078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:11.893078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:11.893078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:11.893078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:11.893078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:11.893078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:11.893078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:11.893078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:11.893078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:11.893078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:11.893078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:11.893428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:11.893428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:11.893428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.893428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:11.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:11.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:11.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:11.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:11.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:11.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:11.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:11.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:11.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:11.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:11.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:11.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:11.894423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:11.894930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:11.894930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:11.894930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:11.894930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:11.894930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:11.894930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.894930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:11.918999 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:11.919699 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:11.919699 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:11.919699 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:11.920017 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:11.920238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.922605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:11.923089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:11.923089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:11.923089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:11.923089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:11.923089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:11.923089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:11.923089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:11.923089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:11.923089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:11.923089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:11.923089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:11.923089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:11.923089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.923089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:11.923089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.923575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:11.923575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:11.923575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:11.923575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.923575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:11.923575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:11.923575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:11.923575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:11.923575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.923575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:11.923575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:11.923575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:11.923575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.923830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:11.923830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:11.923830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:11.923830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:11.923830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:11.923830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:11.923830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:11.923830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:11.923830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:11.923830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:11.923830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:11.923830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:11.923830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:11.924082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.924082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:11.924082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.924082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:11.924082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:11.924082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:11.924082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:11.924082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:11.924082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.924082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:11.924082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:11.924082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.924082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:11.924082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:11.924082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:11.924448 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:11.924825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:11.924825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:11.924825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:11.924825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:11.924825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:11.924825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:11.924825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.924825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:11.964746 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:11.965048 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:11.965195 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:11.965352 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:11.965701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:11.965898 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:11.966158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.966652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:11.966652 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:11.966652 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:11.966652 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:11.966652 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:11.966652 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:11.966652 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.966652 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:11.967400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.967400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:11.967400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:11.967400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:11.967400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.967400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:11.967400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:11.967400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.967400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:11.967400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:11.967400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:11.967400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.967701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:11.967701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:11.967701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:11.967701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:11.967701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:11.967701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:11.967701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:11.967701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:11.967701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:11.967701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:11.967701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:11.967701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:11.967701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:11.967984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.967984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:11.968199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.968199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:11.968199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:11.968199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:11.968199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:11.968199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:11.968199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.968199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:11.968199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:11.968199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.968199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:11.968199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:11.968199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.968199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:11.968199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:11.968199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:11.968526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:11.968906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:11.968906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:11.968906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:11.977986 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:11.978228 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:11.978336 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:11.979160 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:12.017389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:12.017616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:12.018246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:12.018246 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:12.018246 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:12.018246 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:12.018246 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:12.018246 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:12.018246 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:12.018246 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:12.018873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:12.018873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:12.018873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:12.018873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:12.018873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:12.018873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:12.018873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:12.018873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:12.018873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:12.018873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:12.018873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:12.018873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:12.019118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:12.019118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:12.019118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:12.019118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:12.019118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:12.019118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:12.019118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:12.019118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:12.019118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:12.019118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:12.019118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:12.019118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:12.019118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:12.019343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:12.019343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:12.019609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:12.019609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:12.019609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:12.019609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:12.019609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:12.019609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:12.019609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:12.019609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:12.019609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:12.019609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:12.019609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:12.019609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:12.019609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:12.019609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:12.019609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:12.019609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:12.019977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:12.020351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:12.020351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:12.020351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:16.846185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:16.848962 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:16.849191 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:16.850854 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:16.851483 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:16.853274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:16.853274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-85e3f139-41d9-4927-a78c-7fe08bc57f0c, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:16.853274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:16.853274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:16.853274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:16.853672 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:16.853672 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:16.853672 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:16.853672 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:16.853672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.856278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:16.856278 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:16.856278 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:16.856278 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:16.856278 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:16.856278 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:16.856278 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.856278 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:16.858187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.858187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:16.858187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:16.858187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:16.858187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.858187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:16.858187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:16.858187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.858187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:16.858187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:16.858187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:16.858187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.859083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:16.859083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:16.859083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:16.859083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:16.859083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:16.859083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:16.859083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:16.859083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:16.859083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:16.859083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:16.859083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:16.859083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:16.859083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:16.860328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.860328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:16.860328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.860328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:16.860328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:16.860328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:16.860328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:16.860328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:16.860328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.860328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:16.860328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:16.860328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.860328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:16.860328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:16.860328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:16.860959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:16.862329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:16.862329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:16.862329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:16.862329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:16.862329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.862329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:16.887553 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:16.887875 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:16.887982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:16.888516 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:37:16.888828 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:16.888994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.889574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:16.889574 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:16.889574 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:16.889574 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:16.889574 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:16.889574 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:16.889574 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.889574 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:16.890231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.890231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:16.890231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:16.890231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:16.890231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.890231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:16.890231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:16.890231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:16.890231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.890231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:16.890231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:16.890231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:16.890231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:16.890231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:16.890231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:16.891159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:16.891159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:16.891159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:16.891159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:16.891159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:16.891159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:16.891159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:16.891159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.891159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:16.891159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.891159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:16.891159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:16.891159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:16.891159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.891917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:16.891917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:16.891917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:37:16.891917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:16.891917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:16.891917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.891917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:16.891917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:16.891917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:16.891917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.891917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:16.891917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:16.891917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:16.892285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:16.892285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:16.892285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:16.892285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:16.892285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:16.892285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:16.892285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:16.892285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:16.892285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:16.892285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:16.892285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.892285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:16.892490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.892490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:16.892490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:16.892490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:16.892490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:16.892490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:16.892490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.892490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:16.892490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:16.892490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.892490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:16.892490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:16.892490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.892490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:16.892490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:16.892490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:16.892798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:16.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:16.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:16.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:16.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:16.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:16.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:16.919464 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:16.919987 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:16.920071 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:16.920208 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:16.920688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:16.920967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.921665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:16.921665 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:16.921665 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:16.921665 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:16.921665 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:16.921665 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:16.921665 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.921665 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:16.922632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.922632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:16.922632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:16.922632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:16.922632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.922632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:16.922632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:16.922632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:16.922632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.922632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:16.922632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:16.922632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:16.922632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:16.922632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:16.922632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:16.923142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:16.923142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:16.923142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:16.923142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:16.923142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:16.923142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:16.923142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:16.923142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.923142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:16.923709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.923709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:16.923709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:16.923709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:16.923709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.923709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:16.923709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:16.923709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:16.923709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:16.923709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.923709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:16.924101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:16.924101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:16.924101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.924101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:16.924101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:16.924101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:16.924101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:16.924101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:16.924101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:16.924101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:16.924101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:16.924101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:16.924101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:16.924101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:16.924640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:16.924640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:16.924640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.924640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.925175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:16.925631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:16.926374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:16.926374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:16.926374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.926374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:16.965289 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:16.965753 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:16.965893 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:16.966896 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:16.966961 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:16.967285 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:16.967629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.968827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:16.968827 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:16.968827 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:16.968827 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:16.968827 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:16.968827 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:16.968827 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.968827 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:16.969796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.969796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:16.969796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:16.969796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:16.969796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.969796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:16.969796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:16.969796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.969796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:16.969796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:16.969796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:16.969796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.970311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:16.970311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:16.970311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:16.970311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:16.970311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:16.970311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:16.970311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:16.970311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:16.970311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:16.970311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:16.970311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:16.970311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:16.970311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:16.970819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.970819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:16.973273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.973273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:16.973273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:16.973273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:16.973273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:16.973273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:16.973273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.973273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:16.973273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:16.973273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.973273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:16.973273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:16.973273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.973273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:16.973273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:16.973273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:16.973668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:16.974779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:16.974779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:16.974779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:16.978689 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:16.981341 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:16.981801 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:16.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:17.060360 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:17.060546 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:17.060965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:17.060965 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:17.060965 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:17.060965 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:17.060965 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:17.060965 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:17.060965 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:17.060965 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:17.061633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.079 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:17.061633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:17.061633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:17.061633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:17.061633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:17.061633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:17.061633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:17.061633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:17.061633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:17.061633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:17.061633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:17.061633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:17.062965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:17.062965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:17.062965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:17.062965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:17.062965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:17.062965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:17.062965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:17.062965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:17.062965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:17.062965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:17.062965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:17.062965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:17.062965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:17.063698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:17.063698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:17.063698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:17.063698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:17.063698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:17.063698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:17.063698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:17.063698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:17.063698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:17.063698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:17.063698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:17.063698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:17.063698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:17.063698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:17.063698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:17.064249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:17.065110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:17.065110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:17.065110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:17.065110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:17.065110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:17.065110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:21.849325 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:21.849325 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:21.849972 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:21.850554 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:21.850985 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:21.851211 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:21.851619 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:21.851619 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:21.851789 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:21.851961 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:21.852155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:21.852496 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:21.852583 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:21.852729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.854346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:21.854346 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:21.854346 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:21.854346 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:21.854346 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:21.854346 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:21.854346 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.854346 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:21.855215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.855215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:21.855215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:21.855215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:21.855215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.855215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:21.855215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:21.855215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.855215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:21.855215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:21.855215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:21.855215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.855473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:21.855473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:21.855473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:21.855473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:21.855473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:21.855473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:21.855473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:21.855473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:21.855473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:21.855473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:21.855473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:21.855473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:21.855473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:21.855767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.855767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:21.855929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.855929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:21.855929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:21.855929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:21.855929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:21.855929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:21.855929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.855929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:21.855929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:21.855929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.855929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:21.855929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:21.855929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.855929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:21.855929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:21.855929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:21.856227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:21.856724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:21.856724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.856724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:21.888097 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:21.890092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:21.890189 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:21.890277 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:37:21.891997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:21.892452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.892842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:21.892842 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:21.892842 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:21.892842 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:21.892842 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:21.892842 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:21.892842 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.892842 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:21.893425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.893425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:21.893425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:21.893425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:21.893425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.893425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:21.893425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:21.893425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:21.893425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.893425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:21.893425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:21.893425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:21.893425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:21.893425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:21.893425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:21.893855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:21.893855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:21.893855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:21.893855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:21.893855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:21.893855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:21.893855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:21.893855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.893855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:21.894072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.894072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:21.894072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:21.894072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:21.894072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.894072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:21.894072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:21.894072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:37:21.894072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:21.894072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:21.894072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.894072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:21.894374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:21.894374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:21.894374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.894374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:21.894374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:21.894374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:21.894374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:21.894374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:21.894374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:21.894374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:21.894374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:21.894374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:21.894374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:21.894374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:21.894744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:21.894744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:21.894744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.894744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:21.894744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.894744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:21.894744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:21.894744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:21.894744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:21.894744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:21.894744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.894744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:21.894744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:21.894744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:21.895071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:21.895512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:21.895512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:21.895512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:21.895512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:21.895512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:21.895512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:21.895512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:21.895512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:21.895512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:21.895512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:21.895512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.895512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:21.920094 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:21.920736 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:21.920960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:21.921578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:21.922043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.922676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:21.922676 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:21.922676 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:21.922676 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:21.922676 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:21.922676 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:21.922676 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.922676 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:21.923425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.923425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:21.923425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:21.923425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:21.923425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.923425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:21.923425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:21.923425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:21.923425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.923425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:21.923425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:21.923425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:21.923425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:21.923425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:21.923425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:21.924042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:21.924042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:21.924042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:21.924042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:21.924042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:21.924042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:21.924042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:21.924042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.924042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:21.924380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.924380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:21.924380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:21.924380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:21.924380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.924380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:21.924380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:21.924380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:21.924380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:21.924380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.924380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:21.924915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:21.924915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:21.924915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.924915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:21.924915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:21.924915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:21.924915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:21.924915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:21.924915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:21.924915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:21.924915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:21.924915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:21.924915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:21.924915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:21.925163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:21.925163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:21.925163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.925163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.925242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:21.925551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:21.925885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:21.925885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:21.925885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.925885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:21.965850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:21.966637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:21.966637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:21.966637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:21.966785 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:21.967043 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:21.967414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.967984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:21.967984 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:21.967984 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:21.967984 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:21.967984 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:21.967984 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:21.967984 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.967984 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:21.968831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.968831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:21.968831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:21.968831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:21.968831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.968831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:21.968831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:21.968831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.968831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:21.968831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:21.968831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:21.968831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.969185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:21.969185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:21.969185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:21.969185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:21.969185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:21.969185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:21.969185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:21.969185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:21.969185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:21.969185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:21.969185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:21.969185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:21.969185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:21.969381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.969381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:21.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:21.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:21.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:21.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:21.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:21.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:21.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:21.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:21.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:21.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:21.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:21.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:21.969716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:21.970004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:21.970004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:21.970004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:21.981330 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:21.981747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:21.981747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:21.982558 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:21.983694 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=236 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:37:22.289200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:22.290862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:22.290862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:22.290862 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:22.290862 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:22.290862 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:22.290862 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:22.290862 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:22.290862 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:22.290862 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:22.291599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.309 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:22.291599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:22.291599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:22.291599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:22.291599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:22.291599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:22.291599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:22.291599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:22.291599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:22.291599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:22.291599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:22.291599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:22.291995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:22.291995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:22.291995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:22.291995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:22.291995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:22.291995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:22.291995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:22.291995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:22.291995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:22.291995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:22.291995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:22.291995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:22.291995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:22.292313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:22.292313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:22.292840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:22.292840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:22.292840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:22.292840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:22.292840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:22.292840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:22.292840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:22.292840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:22.292840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:22.292840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:22.292840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:22.292840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:22.292840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:22.292840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:22.292840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:22.292840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:22.293334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:22.293931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:22.293931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:22.293931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:26.849938 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:26.850381 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:26.850863 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:26.851946 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:26.852565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:26.854305 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:26.854305 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:26.854305 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:26.854305 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:26.854305 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:26.854455 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:26.854455 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:26.854455 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:26.854455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.855946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:26.855946 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:26.855946 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:26.855946 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:26.855946 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:26.855946 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:26.855946 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.855946 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:26.857822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.857822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:26.857822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:26.857822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:26.857822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.857822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:26.857822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:26.857822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.857822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:26.857822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:26.857822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:26.857822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.858176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:26.858176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:26.858176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:26.858176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:26.858176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:26.858176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:26.858176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:26.858176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:26.858176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:26.858176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:26.858176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:26.858176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:26.858176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:26.858506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.858506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:26.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:26.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:26.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:26.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:26.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:26.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:26.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:26.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:26.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:26.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:26.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:26.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:26.860717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:26.861210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:26.861210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.861210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:26.889649 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:26.890470 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:26.890680 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:26.890855 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:37:26.891924 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:26.891924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.892632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:26.892632 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:26.892632 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:26.892632 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:26.892632 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:26.892632 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:26.892632 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.892632 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:26.893651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.893651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:26.893651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:26.893651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:26.893651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.893651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:26.893651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:26.893651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:26.893651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.893651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:26.893651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:26.893651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:26.893651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:26.893651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:26.893651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:26.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:26.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:26.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:26.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:26.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:26.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:26.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:26.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:26.895271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.895271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:26.895271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:26.895271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:26.895271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.895271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:26.895271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:26.895271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:37:26.895271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:26.895271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:26.895271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.895271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:26.895636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:26.895636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:26.895636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.895636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:26.895636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:26.895636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:26.895636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:26.895636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:26.895636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:26.895636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:26.895636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:26.895636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:26.895636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:26.895636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:26.895987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:26.895987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:26.895987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.895987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:26.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:26.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:26.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:26.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:26.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:26.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:26.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:26.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:26.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:26.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:26.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:26.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:26.897408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:26.897899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:26.897899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:26.897899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:26.897899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:26.897899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:26.897899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.897899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:26.920845 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:26.921271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:26.921524 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:26.921524 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:26.923184 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:26.923184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.924821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:26.925220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:26.925220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:26.925220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:26.925220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:26.925220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:26.925220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:26.925220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:26.925220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:26.925220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:26.925220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:26.925220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:26.925220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:26.925220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.925220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:26.925472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.925472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:26.925472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:26.925472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:26.925472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.925472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:26.925472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:26.925472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:26.925472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:26.925472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.925472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:26.925699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:26.925699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:26.925699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.925699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:26.925699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:26.925699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:26.925699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:26.925699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:26.925699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:26.925699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:26.925699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:26.925699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:26.925699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:26.925699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:26.926244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:26.926244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:26.926244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.926244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.927665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:26.928309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:26.928846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:26.928846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:26.928846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.928846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:26.966816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:26.967784 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:26.968013 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:26.968261 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:26.970091 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:26.970091 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:26.970091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.970091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:26.970091 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:26.970091 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:26.970091 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:26.970091 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:26.970091 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:26.970091 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.970091 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:26.971229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.971229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:26.971229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:26.971229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:26.971229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.971229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:26.971229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:26.971229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.971229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:26.971229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:26.971229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:26.971229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.971663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:26.971663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:26.971663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:26.971663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:26.971663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:26.971663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:26.971663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:26.971663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:26.971663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:26.971663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:26.971663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:26.971663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:26.971663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:26.972148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.972148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:26.972331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.972331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:26.972331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:26.972331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:26.972331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:26.972331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:26.972331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.972331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:26.972331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:26.972331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.972331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:26.972331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:26.972331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.972331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:26.972331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:26.972331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:26.973275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:26.973648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:26.973648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:26.973648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:26.981862 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:26.982440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:26.982792 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:26.984527 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:27.041615 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:27.041615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:27.041808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:27.041808 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:27.041808 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:27.041808 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:27.041808 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:27.041808 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:27.041808 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:27.041808 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:27.042744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.059 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:27.042744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:27.042744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:27.042744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:27.042744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:27.042744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:27.042744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:27.042744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:27.042744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:27.042744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:27.042744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:27.042744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:27.043135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:27.043135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:27.043135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:27.043135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:27.043135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:27.043135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:27.043135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:27.043135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:27.043135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:27.043135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:27.043135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:27.043135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:27.043135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:27.043513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:27.043513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:27.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:27.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:27.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:27.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:27.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:27.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:27.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:27.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:27.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:27.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:27.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:27.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:27.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:27.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:27.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:27.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:27.044300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:27.044711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:27.044711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:27.044711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:31.851648 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:31.851648 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:31.851648 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:31.852117 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:31.852233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:31.852379 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:31.852528 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:31.852664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-9ff629f2-01ec-47a1-95ca-3f1ad0dd13c3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:31.852784 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:31.852894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:31.853001 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:31.853104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:31.853205 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:31.853308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.853866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:31.853866 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:31.853866 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:31.853866 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:31.853866 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:31.853866 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:31.853866 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.853866 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:31.854887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.854887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:31.854887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:31.854887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:31.854887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.854887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:31.854887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:31.854887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.854887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:31.854887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:31.854887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:31.854887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:31.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:31.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:31.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:31.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:31.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:31.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:31.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:31.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:31.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:31.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:31.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:31.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:31.855378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.855378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:31.856288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.856288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:31.856288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:31.856288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:31.856288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:31.856288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:31.856288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.856288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:31.856288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:31.856288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.856288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:31.856288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:31.856288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.856288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:31.856288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:31.856288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:31.856638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:31.856959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:31.856959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.856959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:31.890146 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:31.890642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:31.890796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:31.890947 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:37:31.892009 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:31.892220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.892766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:31.892766 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:31.892766 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:31.892766 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:31.892766 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:31.892766 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:31.892766 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.892766 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:31.893726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.893726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:31.893726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:31.893726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:31.893726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.893726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:31.893726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:31.893726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:31.893726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.893726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:31.893726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:31.893726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:31.893726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:31.893726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:31.893726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:31.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:31.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:31.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:31.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:31.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:31.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:31.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:31.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.894054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:31.894701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.894701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:31.894701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:31.894701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:31.894701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.894701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:31.894701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:31.894701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:37:31.894701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:31.894701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:31.894701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.894701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:31.894941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:31.894941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:31.894941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.894941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:31.894941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:31.894941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:31.894941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:31.894941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:31.894941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:31.894941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:31.894941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:31.894941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:31.894941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:31.894941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:31.895183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:31.895183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:31.895183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.895183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:31.896014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.896014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:31.896014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:31.896014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:31.896014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:31.896014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:31.896014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.896014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:31.896014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:31.896014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.896014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:31.896014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:31.896014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.896014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:31.896014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:31.896014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:31.896323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:31.896652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:31.896652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:31.896652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:31.896652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:31.896652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:31.896652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.896652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:31.921254 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:31.921676 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:31.921770 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:31.921874 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:31.922996 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:31.923211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.923787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:31.923787 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:31.923787 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:31.923787 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:31.923787 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:31.923787 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:31.923787 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.923787 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:31.924531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.924531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:31.924531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:31.924531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:31.924531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.924531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:31.924531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:31.924531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:31.924531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.924531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:31.924531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:31.924531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:31.924531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:31.924531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:31.924531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:31.924871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:31.924871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:31.924871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:31.924871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:31.924871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:31.924871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:31.924871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:31.924871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.924871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:31.926423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.926423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:31.926423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:31.926423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:31.926423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.926423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:31.926423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:31.926423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:31.926423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:31.926423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.926423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:31.926855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:31.926855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:31.926855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.926855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:31.926855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:31.926855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:31.926855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:31.926855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:31.926855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:31.926855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:31.926855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:31.926855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:31.926855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:31.926855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:31.927191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:31.927191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:31.927191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.927191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:31.927191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.927191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:31.927191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:31.927191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:31.927191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:31.927191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:31.927191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.927191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:31.927191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:31.927191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:31.927516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:31.928132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:31.928132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:31.928132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:31.928132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:31.928132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:31.928132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:31.928132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:31.928132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:31.928132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:31.928132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.928132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:31.967396 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:31.967819 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:31.968043 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:31.968043 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:31.968405 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:31.968674 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:31.969031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.969506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:31.969506 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:31.969506 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:31.969506 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:31.969506 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:31.969506 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:31.969506 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.969506 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:31.970296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.970296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:31.970296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:31.970296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:31.970296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.970296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:31.970296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:31.970296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.970296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:31.970296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:31.970296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:31.970296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.970630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:31.970630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:31.970630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:31.970630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:31.970630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:31.970630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:31.970630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:31.970630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:31.970630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:31.970630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:31.970630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:31.970630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:31.970630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:31.970919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.970919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:31.971425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.971425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:31.971425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:31.971425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:31.971425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:31.971425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:31.971425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.971425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:31.971425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:31.971425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.971425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:31.971425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:31.971425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.971425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:31.971425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:31.971425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:31.971797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:31.972401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:31.972401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:31.972401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:31.982413 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:31.982960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:31.982960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:31.983694 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:32.016747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:32.017098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:32.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:32.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:32.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:32.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:32.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:32.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:32.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:32.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:32.018368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:32.018368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:32.018368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:32.018368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:32.018368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:32.018368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:32.018368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:32.018368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:32.018368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:32.018368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:32.018368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:32.018368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:32.018692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:32.018692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:32.018692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:32.018692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:32.018692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:32.018692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:32.018692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:32.018692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:32.018692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:32.018692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:32.018692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:32.018692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:32.018692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:32.018968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:32.018968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:32.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:32.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:32.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:32.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:32.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:32.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:32.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:32.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:32.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:32.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:32.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:32.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:32.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:32.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:32.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:32.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:32.019457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:32.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:32.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:32.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:36.850954 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:36.851678 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:36.851678 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:36.852197 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:36.852403 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:36.852622 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:36.852778 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:36.852964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:36.853116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:36.853270 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:36.853436 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:36.853598 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:36.853707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.854221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:36.854221 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:36.854221 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:36.854221 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:36.854221 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:36.854221 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:36.854221 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.854221 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:36.855023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.855023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:36.855023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:36.855023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:36.855023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.855023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:36.855023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:36.855023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.855023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:36.855023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:36.855023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:36.855023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.855373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:36.855373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:36.855373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:36.855373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:36.855373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:36.855373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:36.855373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:36.855373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:36.855373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:36.855373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:36.855373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:36.855373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:36.855373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:36.855719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.855719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:36.856285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.856285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:36.856285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:36.856285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:36.856285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:36.856285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:36.856285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.856285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:36.856285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:36.856285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.856285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:36.856285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:36.856285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.856285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:36.856285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:36.856285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:36.856698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:36.857049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:36.857049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.857049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:36.890694 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:36.890901 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:36.891023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:36.891167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:37:36.891550 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:36.891752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.892258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:36.892258 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:36.892258 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:36.892258 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:36.892258 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:36.892258 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:36.892258 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.892258 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:36.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:36.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:36.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:36.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:36.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:36.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:36.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:36.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:36.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:36.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:36.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:36.893111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:36.893397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:36.893397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:36.893397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:36.893397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:36.893397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:36.893397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:36.893397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:36.893397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.893397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:36.894000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.894000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:36.894000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:36.894000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:36.894000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.894000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:36.894000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:36.894000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:37:36.894000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:36.894000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:36.894000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.894000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:36.894304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:36.894304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:36.894304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.894304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:36.894304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:36.894304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:36.894304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:36.894304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:36.894304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:36.894304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:36.894304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:36.894304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:36.894304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:36.894304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:36.894574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:36.894574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:36.894574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.894574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:36.895398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:36.895726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:36.895726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:36.895726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:36.895726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:36.895726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:36.895726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.895726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:36.921804 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:36.922064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:36.922138 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:36.922262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:36.922630 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:36.922821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.923251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:36.923251 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:36.923251 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:36.923251 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:36.923251 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:36.923251 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:36.923251 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.923251 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:36.924097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.924097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:36.924097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:36.924097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:36.924097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.924097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:36.924097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:36.924097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:36.924097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.924097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:36.924097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:36.924097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:36.924097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:36.924097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:36.924097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:36.924375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:36.924375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:36.924375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:36.924375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:36.924375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:36.924375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:36.924375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:36.924375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.924375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:36.925093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.925093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:36.925093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:36.925093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:36.925093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.925093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:36.925093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:36.925093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:36.925093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:36.925093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.925093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:36.925304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:36.925304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:36.925304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.925304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:36.925304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:36.925304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:36.925304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:36.925304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:36.925304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:36.925304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:36.925304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:36.925304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:36.925304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:36.925304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:36.925577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:36.925577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:36.925577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.925577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.926249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:36.926672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:36.927014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:36.927014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:36.927014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.927014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:36.967818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:36.968055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:36.968144 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:36.968252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:36.968700 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:36.968920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:36.969218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.969701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:36.969701 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:36.969701 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:36.969701 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:36.969701 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:36.969701 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:36.969701 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.969701 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:36.970466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.970466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:36.970466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:36.970466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:36.970466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.970466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:36.970466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:36.970466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.970466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:36.970466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:36.970466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:36.970466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.970726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:36.970726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:36.970726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:36.970726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:36.970726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:36.970726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:36.970726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:36.970726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:36.970726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:36.970726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:36.970726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:36.970726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:36.970726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:36.970964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.970964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:36.971225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.971225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:36.971225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:36.971225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:36.971225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:36.971225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:36.971225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.971225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:36.971225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:36.971225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.971225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:36.971225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:36.971225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.971225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:36.971225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:36.971225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:36.971477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:36.971808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:36.971808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:36.971808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:36.982734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:36.982931 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:36.983050 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:36.983657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:37.013844 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:37.014064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:37.014442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:37.014442 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:37.014442 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:37.014442 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:37.014442 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:37.014442 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:37.014442 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:37.014442 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:37.015249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:37.015249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:37.015249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:37.015249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:37.015249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:37.015249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:37.015249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:37.015249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:37.015249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:37.015249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:37.015249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:37.015249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:37.015596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:37.015596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:37.015596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:37.015596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:37.015596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:37.015596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:37.015596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:37.015596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:37.015596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:37.015596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:37.015596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:37.015596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:37.015596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:37.015811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:37.015811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:37.016142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:37.016142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:37.016142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:37.016142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:37.016142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:37.016142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:37.016142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:37.016142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:37.016142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:37.016142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:37.016142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:37.016142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:37.016142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:37.016142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:37.016142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:37.016142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:37.016925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:37.017276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:37.017276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:37.017276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:41.851368 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:41.851945 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:41.851945 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:41.852802 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:41.853022 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:41.853187 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:41.853307 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:41.853573 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:41.853722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:41.853875 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:41.853978 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:41.854080 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:41.854186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.854697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:41.854697 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:41.854697 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:41.854697 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:41.854697 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:41.854697 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:41.854697 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.854697 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:41.855578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.855578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:41.855578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:41.855578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:41.855578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.855578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:41.855578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:41.855578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.855578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:41.855578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:41.855578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:41.855578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.855934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:41.855934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:41.855934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:41.855934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:41.855934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:41.855934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:41.855934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:41.855934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:41.855934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:41.855934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:41.855934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:41.855934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:41.855934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:41.856292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.856292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:41.856761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.856761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:41.856761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:41.856761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:41.856761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:41.856761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:41.856761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.856761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:41.856761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:41.856761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.856761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:41.856761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:41.856761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.856761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:41.856761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:41.856761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:41.857094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:41.857438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:41.857438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.857438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:41.891003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:41.891247 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:41.891338 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:41.891417 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:37:41.891913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:41.892115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.892590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:41.892590 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:41.892590 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:41.892590 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:41.892590 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:41.892590 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:41.892590 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.892590 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:41.893186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.893186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:41.893186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:41.893186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:41.893186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.893186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:41.893186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:41.893186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:41.893186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.893186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:41.893186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:41.893186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:41.893186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:41.893186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:41.893186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:41.893465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:41.893465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:41.893465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:41.893465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:41.893465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:41.893465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:41.893465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:41.893465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.893465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:41.894167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.894167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:41.894167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:41.894167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:41.894167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.894167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:41.894167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:41.894167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:37:41.894167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:41.894167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:41.894167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.894167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:41.894655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:41.894655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:41.894655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.894655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:41.894655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:41.894655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:41.894655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:41.894655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:41.894655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:41.894655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:41.894655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:41.894655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:41.894655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:41.894655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:41.894885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:41.894885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:41.894885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.894885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:41.894885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.894885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:41.894885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:41.894885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:41.894885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:41.894885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:41.894885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.894885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:41.894885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:41.894885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:41.895117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:41.895400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:41.895400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:41.895400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:41.895400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:41.895400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:41.895400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:41.895400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:41.895400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:41.895400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:41.895400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:41.895400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.895400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:41.922116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:41.922418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:41.922583 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:41.922848 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:41.923323 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:41.923547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.923980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:41.923980 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:41.923980 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:41.923980 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:41.923980 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:41.923980 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:41.923980 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.923980 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:41.924759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.924759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:41.924759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:41.924759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:41.924759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.924759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:41.924759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:41.924759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:41.924759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.924759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:41.924759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:41.924759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:41.924759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:41.924759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:41.924759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:41.925126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:41.925126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:41.925126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:41.925126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:41.925126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:41.925126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:41.925126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:41.925126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.925126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:41.925709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.925709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:41.925709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:41.925709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:41.925709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.925709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:41.925709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:41.925709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:41.925709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:41.925709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.925709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:41.926035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:41.926035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:41.926035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.926035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:41.926035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:41.926035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:41.926035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:41.926035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:41.926035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:41.926035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:41.926035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:41.926035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:41.926035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:41.926035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:41.926353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:41.926353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:41.926353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.926353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.926941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:41.927245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:41.927573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:41.927573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:41.927573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.927573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:41.968297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:41.968599 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:41.968666 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:41.968881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:41.969200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:41.969395 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:41.969692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.970140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:41.970140 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:41.970140 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:41.970140 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:41.970140 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:41.970140 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:41.970140 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.970140 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:41.970773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.970773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:41.970773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:41.970773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:41.970773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.970773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:41.970773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:41.970773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.970773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:41.970773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:41.970773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:41.970773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.970990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:41.970990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:41.970990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:41.970990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:41.970990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:41.970990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:41.970990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:41.970990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:41.970990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:41.970990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:41.970990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:41.970990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:41.970990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:41.971313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.971313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:41.971634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.971634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:41.971634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:41.971634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:41.971634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:41.971634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:41.971634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.971634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:41.971634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:41.971634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.971634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:41.971634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:41.971634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.971634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:41.971634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:41.971634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:41.972455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:41.972975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:41.972975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:41.972975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:41.983099 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:41.983341 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:41.983418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:41.984047 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:42.022027 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:42.022296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:42.022867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:42.022867 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:42.022867 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:42.022867 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:42.022867 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:42.022867 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:42.022867 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:42.022867 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:42.023498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:42.023498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:42.023498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:42.023498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:42.023498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:42.023498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:42.023498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:42.023498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:42.023498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:42.023498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:42.023498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:42.023498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:42.023821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:42.023821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:42.023821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:42.023821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:42.023821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:42.023821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:42.023821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:42.023821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:42.023821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:42.023821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:42.023821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:42.023821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:42.023821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:42.024042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:42.024042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:42.024320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:42.024320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:42.024320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:42.024320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:42.024320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:42.024320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:42.024320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:42.024320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:42.024320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:42.024320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:42.024320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:42.024320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:42.024320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:42.024320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:42.024320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:42.024320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:42.025210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:42.025594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:42.025594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:42.025594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:46.851885 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:46.852340 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:46.852402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:46.854281 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:46.854454 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:46.854604 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:46.854717 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:46.854821 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:46.854920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:46.855017 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:46.855114 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-be337b90-4d3e-4762-901d-a630bf921543, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:46.855211 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:46.855313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.855925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:46.855925 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:46.855925 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:46.855925 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:46.855925 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:46.855925 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:46.855925 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.855925 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:46.856877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.856877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:46.856877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:46.856877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:46.856877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.856877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:46.856877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:46.856877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.856877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:46.856877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:46.856877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:46.856877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.857124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:46.857124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:46.857124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:46.857124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:46.857124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:46.857124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:46.857124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:46.857124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:46.857124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:46.857124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:46.857124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:46.857124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:46.857124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:46.857335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.857335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:46.861178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.861178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:46.861178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:46.861178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:46.861178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:46.861178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:46.861178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.861178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:46.861178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:46.861178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.861178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:46.861178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:46.861178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.861178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:46.861178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:46.861178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:46.861652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:46.862015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:46.862015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.862015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:46.891444 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:46.892388 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:46.892388 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:46.892388 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:37:46.892914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:46.893088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.894585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:46.894585 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:46.894585 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:46.894585 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:46.894585 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:46.894585 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:46.894585 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.894585 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:46.897564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.897564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:46.897564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:46.897564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:46.897564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.897564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:46.897564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:46.897564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:46.897564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.897564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:46.897564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:46.897564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:46.897564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:46.897564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:46.897564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:46.898110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:46.898110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:46.898110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:46.898110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:46.898110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:46.898110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:46.898110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:46.898110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.898110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:46.898110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.898110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:46.898110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:46.898110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:46.898110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.898609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:46.898609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:46.898609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:37:46.898609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:46.898609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:46.898609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.898609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:46.898609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:46.898609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:46.898609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.898609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:46.898609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:46.898609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:46.899085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:46.899085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:46.899085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:46.899085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:46.899085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:46.899085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:46.899085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:46.899085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:46.899085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:46.899085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:46.899085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.899085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:46.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:46.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:46.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:46.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:46.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:46.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:46.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:46.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:46.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:46.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:46.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:46.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:46.901196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:46.902024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:46.902024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:46.902024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:46.902024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:46.902024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:46.902024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.902024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:46.922551 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:46.923182 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:46.923182 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:46.923182 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:46.923627 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:46.923828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.926294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:46.926720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:46.926720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:46.926720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:46.926720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:46.926720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:46.926720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:46.926720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:46.926720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:46.926720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:46.926720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:46.926720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:46.926720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:46.926720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.926720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:46.926720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.926988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:46.926988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:46.926988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:46.926988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.926988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:46.926988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:46.926988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:46.926988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:46.926988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.926988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:46.926988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:46.926988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:46.926988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.927224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:46.927224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:46.927224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:46.927224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:46.927224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:46.927224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:46.927224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:46.927224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:46.927224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:46.927224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:46.927224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:46.927224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:46.927224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:46.927513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.927513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:46.927513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.927513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:46.927513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:46.927513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:46.927513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:46.927513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:46.927513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.927513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:46.927513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:46.927513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.927513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:46.927513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:46.927513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:46.927784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:46.928131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:46.928131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:46.928131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:46.928131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:46.928131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:46.928131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:46.928131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.928131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:46.968807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:46.969816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:46.969816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:46.969816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:46.988664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:46.990577 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:46.990823 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:46.993777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.994952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:46.994952 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:46.994952 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:46.994952 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:46.994952 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:46.994952 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:46.994952 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.994952 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:46.996771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.996771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:46.996771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:46.996771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:46.996771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.996771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:46.996771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:46.996771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.996771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:46.996771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:46.996771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:46.996771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.997153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:46.997153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:46.997153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:46.997153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:46.997153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:46.997153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:46.997153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:46.997153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:46.997153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:46.997153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:46.997153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:46.997153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:46.997153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:46.997505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.997505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:46.998779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.998779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:46.998779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:46.998779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:46.998779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:46.998779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:46.998779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.998779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:46.998779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:46.998779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.998779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:46.998779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:46.998779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.998779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:46.998779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:46.998779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:46.999254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:46.999773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:46.999773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:46.999773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:47.000796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:47.000963 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:47.004035 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:47.049256 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:47.049485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:47.052411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:47.052411 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:47.052411 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:47.052411 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:47.052411 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:47.052411 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:47.052411 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:47.052411 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:47.052411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:47.052411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:47.052411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:47.052411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:47.052411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:47.052411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:47.052411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:47.052910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:47.052910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:47.052910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:47.052910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:47.052910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:47.052910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:47.052910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:47.052910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:47.052910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:47.052910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:47.052910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:47.052910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:47.052910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:47.052910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:47.053236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:47.053236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:47.053236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:47.053236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:47.053236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:47.053236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:47.053236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:47.053236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:47.053236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:47.053236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:47.053236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:47.053236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:47.053236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:47.053236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:47.053694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:47.054014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:47.054014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:47.054014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:47.054014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:47.054014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:47.054014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:47.054014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:47.054014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:47.054014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:51.852559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:51.853506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:51.853506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:51.854194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:51.854379 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:51.854567 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:51.854695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:51.854812 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:51.854913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:51.855014 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:51.855115 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:51.855216 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:51.855316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.855818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:51.855818 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:51.855818 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:51.855818 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:51.855818 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:51.855818 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:51.855818 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.855818 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:51.857294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.857294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:51.857294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:51.857294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:51.857294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.857294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:51.857294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:51.857294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.857294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:51.857294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:51.857294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:51.857294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.857627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:51.857627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:51.857627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:51.857627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:51.857627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:51.857627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:51.857627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:51.857627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:51.857627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:51.857627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:51.857627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:51.857627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:51.857627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:51.857947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.857947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:51.859276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.859276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:51.859276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:51.859276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:51.859276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:51.859276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:51.859276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.859276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:51.859276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:51.859276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.859276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:51.859276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:51.859276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.859276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:51.859276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:51.859276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:51.859710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:51.860138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:51.860138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.860138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:51.891933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:51.892314 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:51.893067 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:51.893234 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:37:51.893817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:51.893992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.894748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:51.894748 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:51.894748 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:51.894748 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:51.894748 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:51.894748 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:51.894748 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.894748 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:51.895473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.895473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:51.895473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:51.895473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:51.895473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.895473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:51.895473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:51.895473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:51.895473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.895473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:51.895473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:51.895473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:51.895473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:51.895473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:51.895473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:51.895815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:51.895815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:51.895815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:51.895815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:51.895815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:51.895815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:51.895815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:51.895815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.895815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:51.898316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.898316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:51.898316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:51.898316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:51.898316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.898316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:51.898316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:51.898316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:37:51.898316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:51.898316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:51.898316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.898316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:51.898600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:51.898600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:51.898600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.898600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:51.898600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:51.898600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:51.898600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:51.898600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:51.898600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:51.898600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:51.898600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:51.898600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:51.898600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:51.898600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:51.898852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:51.898852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:51.898852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.898852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:51.899262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.899262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:51.899262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:51.899262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:51.899262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:51.899262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:51.899262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.899262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:51.899262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:51.899262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.899262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:51.899262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:51.899262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.899262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:51.899262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:51.899262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:51.899585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:51.899900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:51.899900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:51.899900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:51.899900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:51.899900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:51.899900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.899900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:51.922983 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:51.928968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:51.929251 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:51.929415 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:51.929802 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:51.931033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.931551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:51.931551 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:51.931551 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:51.931551 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:51.931551 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:51.931551 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:51.931551 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.931551 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:51.932428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.932428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:51.932428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:51.932428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:51.932428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.932428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:51.932428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:51.932428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:51.932428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.932428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:51.932428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:51.932428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:51.932428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:51.932428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:51.932428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:51.932785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:51.932785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:51.932785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:51.932785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:51.932785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:51.932785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:51.932785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:51.932785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.932785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:51.933625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.933625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:51.933625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:51.933625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:51.933625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.933625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:51.933625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:51.933625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:51.933625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:51.933625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.933625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:51.933880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:51.933880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:51.933880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.933880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:51.933880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:51.933880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:51.933880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:51.933880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:51.933880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:51.933880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:51.933880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:51.933880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:51.933880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:51.933880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:51.934135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:51.934135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:51.934135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.934135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.934982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:51.935324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:51.935700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:51.935700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:51.935700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.935700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:51.969390 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:51.969834 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:51.969964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:51.970137 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:51.970481 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:51.970719 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:51.970984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.971507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:51.971507 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:51.971507 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:51.971507 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:51.971507 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:51.971507 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:51.971507 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.971507 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:51.972109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.972109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:51.972109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:51.972109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:51.972109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.972109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:51.972109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:51.972109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.972109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:51.972109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:51.972109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:51.972109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.972367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:51.972367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:51.972367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:51.972367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:51.972367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:51.972367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:51.972367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:51.972367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:51.972367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:51.972367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:51.972367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:51.972367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:51.972367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:51.973771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.973771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:51.973771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.973771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:51.973771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:51.973771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:51.973771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:51.973771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:51.973771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.973771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:51.973771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:51.973771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.973771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:51.973771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:51.973771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:51.974100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:51.974413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:51.974413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:51.974413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:51.974413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:51.974413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:51.974413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:52.000596 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:52.002177 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:52.002334 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:52.003064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:52.054580 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:52.054580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:52.056811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:52.056811 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:52.056811 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:52.056811 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:52.056811 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:52.056811 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:52.056811 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:52.056811 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:52.059393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:52.059393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:52.059393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:52.059393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:52.059393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:52.059393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:52.059393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:52.059393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:52.059393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:52.059393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:52.059393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:52.059393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:52.059749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:52.059749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:52.059749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:52.059749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:52.059749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:52.059749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:52.059749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:52.059749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:52.059749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:52.059749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:52.059749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:52.059749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:52.059749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:52.060003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:52.060003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:52.060003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:52.060003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:52.060003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:52.060003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:52.060003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:52.060003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:52.060003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:52.060003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:52.060003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:52.060003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:52.060003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:52.060003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:52.060003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:52.060351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:52.060802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:52.060802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:52.060802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:52.060802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:52.060802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:52.060802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:56.852851 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:56.853405 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:56.853405 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:56.854607 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:56.854807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:56.854976 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:56.855138 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:56.855266 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:56.855385 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:56.855591 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:56.855630 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:56.855930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:56.855930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:56.855930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.856956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:56.856956 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:56.856956 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:56.856956 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:56.856956 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:56.856956 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:56.856956 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.856956 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:56.858026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.858026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:56.858026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:56.858026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:56.858026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.858026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:56.858026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:56.858026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.858026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:56.858026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:56.858026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:56.858026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.858439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:56.858439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:56.858439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:56.858439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:56.858439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:56.858439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:56.858439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:56.858439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:56.858439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:56.858439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:56.858439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:56.858439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:56.858439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:56.858862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.858862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:56.859822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.859822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:56.859822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:56.859822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:56.859822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:56.859822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:56.859822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.859822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:56.859822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:56.859822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.859822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:56.859822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:56.859822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.859822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:56.859822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:56.859822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:56.860278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:56.860658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:56.860658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.860658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:56.892434 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:56.892828 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:56.892948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:56.893048 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:37:56.893594 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:56.893864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.895193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:56.895193 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:56.895193 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:56.895193 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:56.895193 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:56.895193 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:56.895193 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.895193 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:56.895864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.895864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:56.895864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:56.895864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:56.895864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.895864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:56.895864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:56.895864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:56.895864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.895864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:56.895864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:56.895864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:56.895864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:56.895864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:56.895864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:56.896216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:56.896216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:56.896216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:56.896216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:56.896216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:56.896216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:56.896216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:56.896216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.896216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:56.896447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.896447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:56.896447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:56.896447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:56.896447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.896447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:56.896447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:56.896447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:37:56.896447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:56.896447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:56.896447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.896447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:56.896758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:56.896758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:56.896758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.896758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:56.896758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:56.896758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:56.896758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:56.896758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:56.896758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:56.896758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:56.896758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:56.896758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:56.896758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:56.896758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:56.897035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:56.897035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:56.897035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.897035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:56.897314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.897314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:56.897314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:56.897314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:56.897314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:56.897314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:56.897314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.897314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:56.897314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:56.897314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.897314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:56.897314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:56.897314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.897314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:56.897314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:56.897314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:56.897990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:56.899004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:56.899004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:56.899004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:56.899004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:56.899004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:56.899004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.899004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:56.929021 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:56.929387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:56.929568 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:56.929708 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:56.930386 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:56.930658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.932951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:56.932951 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:56.932951 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:56.932951 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:56.932951 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:56.932951 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:56.932951 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.932951 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:56.933828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.933828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:37:56.933828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:56.933828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:56.933828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.933828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:56.933828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:56.933828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:37:56.933828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.933828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:37:56.933828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:56.933828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:37:56.933828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:56.933828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:37:56.933828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:37:56.934292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:56.934292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:37:56.934292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:56.934292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:37:56.934292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:37:56.934292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:56.934292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:37:56.934292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.934292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:37:56.934794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.934794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:56.934794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:56.934794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:56.934794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.934794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:56.934794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:56.934794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:56.934794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:56.934794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.934794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:56.935464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:56.935464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:56.935464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.935464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:56.935464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:56.935464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:56.935464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:56.935464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:56.935464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:56.935464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:56.935464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:56.935464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:56.935464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:56.935464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:56.936277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:56.936277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:56.936277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.936277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:56.936277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.936277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:56.936277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:56.936277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:56.936277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:56.936277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:56.936277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.936277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:56.936277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:56.936277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:56.936895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:56.937418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:56.937418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:56.937418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:56.937418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:56.937418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:56.937418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:56.937418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:56.937418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:56.937418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:56.937418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.937418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:56.970398 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:56.970942 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:56.971010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:56.971171 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:56.971506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:56.971749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:56.971984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.972430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:56.972430 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:56.972430 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:56.972430 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:56.972430 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:56.972430 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:56.972430 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.972430 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:56.973511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.973511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:56.973511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:56.973511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:56.973511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.973511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:56.973511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:56.973511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.973511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:56.973511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:56.973511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:56.973511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.974046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:56.974046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:56.974046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:56.974046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:56.974046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:56.974046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:56.974046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:56.974046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:56.974046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:56.974046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:56.974046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:56.974046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:56.974046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:56.974272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.974272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:56.974272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.974272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:56.974272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:56.974272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:56.974272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:56.974272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:56.974272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.974272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:56.974272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:56.974272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.974272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:56.974272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:56.974272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:56.974621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:56.974965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:56.974965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:56.974965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:56.974965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:56.974965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:56.974965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:37:57.002283 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:37:57.002702 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:37:57.002817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:37:57.003383 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:37:57.048336 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:37:57.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:57.048889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:37:57.048889 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:57.048889 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:37:57.048889 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:37:57.048889 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:57.048889 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:37:57.048889 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:57.048889 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:37:57.049682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:57.049682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:37:57.049682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:57.049682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:37:57.049682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:57.049682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:57.049682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:57.049682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:57.049682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:57.049682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:57.049682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:37:57.049682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:57.050424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:37:57.050424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:57.050424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:37:57.050424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:57.050424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:37:57.050424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:37:57.050424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:57.050424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:37:57.050424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:57.050424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:37:57.050424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:37:57.050424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:57.050424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:37:57.050857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:57.050857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:37:57.050857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:57.050857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:37:57.050857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:37:57.050857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:37:57.050857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:37:57.050857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:37:57.050857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:57.050857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:37:57.050857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:37:57.050857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:57.050857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:37:57.050857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:37:57.050857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:37:57.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:37:57.051824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:37:57.051824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:37:57.051824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:37:57.051824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:37:57.051824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:37:57.051824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:01.853282 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:01.853891 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:01.853891 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:01.857953 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:01.858278 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:01.858553 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-92e24d25-3017-4fca-a446-c62b0f38822a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:01.858775 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:01.859022 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:01.859281 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:01.859527 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:01.859806 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:01.860029 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:01.860295 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-01dadb77-9ad4-4788-9ddf-e0c79e2c1434, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:01.860562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.861428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:01.861428 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:01.861428 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:01.861428 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:01.861428 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:01.861428 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:01.861428 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.861428 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:01.862833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.862833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:01.862833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:01.862833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:01.862833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.862833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:01.862833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:01.862833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.862833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:01.862833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:01.862833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:01.862833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.863175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:01.863175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:01.863175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:01.863175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:01.863175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:01.863175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:01.863175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:01.863175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:01.863175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:01.863175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:01.863175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:01.863175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:01.863175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:01.863447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.863447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:01.863952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.863952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:01.863952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:01.863952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:01.863952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:01.863952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:01.863952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.863952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:01.863952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:01.863952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.863952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:01.863952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:01.863952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.863952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:01.863952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:01.863952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:01.864274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:01.865261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:01.865261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.865261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:01.895977 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:01.897994 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:01.898155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:01.898312 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:38:01.898934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:01.900020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.900235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:01.900235 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:01.900235 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:01.900235 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:01.900235 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:01.900235 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:01.900235 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.900235 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:01.903626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.903626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:01.903626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:01.903626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:01.903626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.903626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:01.903626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:01.903626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:01.903626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.903626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:01.903626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:01.903626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:01.903626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:01.903626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:01.903626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:01.904107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:01.904107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:01.904107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:01.904107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:01.904107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:01.904107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:01.904107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:01.904107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.904107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:01.904107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.904107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:01.904107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:01.904107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:01.904107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.904436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:01.904436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:01.904436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:38:01.904436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:01.904436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:01.904436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.904436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:01.904436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:01.904436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:01.904436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.904436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:01.904436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:01.904436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:01.904735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:01.904735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:01.904735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:01.904735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:01.904735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:01.904735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:01.904735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:01.904735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:01.904735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:01.904735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:01.904735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.904735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:01.904940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.904940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:01.904940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:01.904940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:01.904940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:01.904940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:01.904940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.904940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:01.904940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:01.904940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.904940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:01.904940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:01.904940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.904940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:01.904940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:01.904940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:01.905307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:01.905823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:01.905823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:01.905823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:01.905823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:01.905823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:01.905823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.905823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:01.930224 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:01.934582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:01.934582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:01.934582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:01.936755 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.938008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:01.938008 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:01.938008 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:01.938008 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:01.938008 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:01.938008 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:01.938008 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.938008 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:01.938748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.938748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:01.938748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:01.938748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:01.938748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.938748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:01.938748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:01.938748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:01.938748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.938748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:01.938748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:01.938748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:01.938748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:01.938748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:01.938748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:01.939204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:01.939204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:01.939204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:01.939204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:01.939204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:01.939204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:01.939204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:01.939204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.939204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:01.939425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.939425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:01.939425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:01.939425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:01.939425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.939425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:01.939425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:01.939425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:01.939425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:01.939425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.939425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:01.940525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:01.940525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:01.940525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.940525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:01.940525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:01.940525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:01.940525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:01.940525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:01.940525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:01.940525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:01.940525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:01.940525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:01.940525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:01.940525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:01.941010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:01.941010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:01.941010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.941010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:01.941010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.941010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:01.941010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:01.941010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:01.941010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:01.941010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:01.941010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.941010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:01.941010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:01.941010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:01.942036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:01.942410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:01.942410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:01.942410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:01.942410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:01.942410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:01.942410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:01.942410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:01.942410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:01.942410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:01.942410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.942410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:01.971066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:01.971403 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:01.971586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:01.971846 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:01.973446 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:01.973446 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:01.973446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.974089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:01.974089 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:01.974089 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:01.974089 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:01.974089 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:01.974089 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:01.974089 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.974089 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:01.974630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.974630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:01.974630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:01.974630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:01.974630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.974630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:01.974630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:01.974630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.974630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:01.974630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:01.974630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:01.974630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.975021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:01.975021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:01.975021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:01.975021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:01.975021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:01.975021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:01.975021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:01.975021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:01.975021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:01.975021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:01.975021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:01.975021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:01.975021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:01.975407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.975407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:01.975549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.975549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:01.975549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:01.975549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:01.975549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:01.975549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:01.975549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.975549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:01.975549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:01.975549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.975549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:01.975549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:01.975549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.975549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:01.975549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:01.975549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:01.976025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:01.976609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:01.976609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:01.976609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:02.003555 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:02.004026 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:02.004026 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:02.005532 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:02.047687 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:02.048128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:02.048994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:02.048994 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:02.048994 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:02.048994 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:02.048994 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:02.048994 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:02.048994 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:02.048994 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:02.050778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:02.050778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:02.050778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:02.050778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:02.050778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:02.050778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:02.050778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:02.050778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:02.050778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:02.050778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:02.050778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:02.050778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:02.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:02.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:02.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:02.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:02.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:02.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:02.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:02.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:02.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:02.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:02.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:02.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:02.051286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:02.051759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:02.051759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:02.054450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:02.054450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:02.054450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:02.054450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:02.054450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:02.054450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:02.054450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:02.054450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:02.054450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:02.054450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:02.054450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:02.054450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:02.054450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:02.054450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:02.054450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:02.054450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:02.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:02.055453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:02.055453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:02.055453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:06.939831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:06.939831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:06.939831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:06.939831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:06.939831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:06.999132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:06.999132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:06.999132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:06.999132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:06.999132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:07.019688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:07.019688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.019688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:07.019688 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.019688 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:07.019688 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:07.019688 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.019688 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:07.019688 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.019688 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:07.019688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.019688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:07.019688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:07.019688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:07.028949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.028949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:07.028949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:07.028949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.028949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:07.028949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:07.028949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:07.028949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.028949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:07.028949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:07.028949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:07.028949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:07.028949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:07.028949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:07.030821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:07.030821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:07.030821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.030821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:07.030821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:07.030821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.030821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:07.030821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.030821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:07.030821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.030821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:07.030821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:07.032834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.035054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:07.035054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:07.035054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:07.035054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:07.035054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:07.035054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:07.035054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:07.035054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:07.035054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.035054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:07.035054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:07.035054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.035054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:07.035054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.035054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:07.035054 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:07.038248 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:07.038248 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:07.038248 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:38:07.038248 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:07.038248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.038248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:07.038248 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.038248 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:07.038248 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:07.039382 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.039382 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:07.039382 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.039382 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:07.039382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.039382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:07.039382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:07.039382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:07.039382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.039382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:07.039382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:07.039382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:07.039382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.039382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:07.039382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:07.039382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:07.043784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:07.043784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:07.043784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:07.043784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:07.043784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:07.043784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.043784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:07.043784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:07.043784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.043784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:07.043784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.043784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:07.043784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.043784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:07.044302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:07.044302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:07.044302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.044302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:07.044302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:07.044302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:38:07.044302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:07.044302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:07.044302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.044302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:07.044302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:07.044302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:07.044302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.045659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:07.045659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:07.045659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:07.045659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:07.045659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:07.045659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:07.045659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:07.045659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:07.045659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.045659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:07.045659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:07.045659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.045659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:07.048709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:07.050508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:07.050508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:07.050508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.050508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:07.050508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:07.050508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.050508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:07.050508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.050508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:07.050508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:07.050508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:07.050508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:07.054132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:07.054132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:07.054132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.054132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:07.054132 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.054132 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:07.054132 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:07.054132 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.054132 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:07.054132 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.054132 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:07.054132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:07.055302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.056743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:07.056743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:07.056743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.056743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:07.056743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.056743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:07.056743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.056743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:07.056743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:07.056743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:07.056743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.056743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:07.056743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:07.059775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:07.059775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:07.059775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.059775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:07.059775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:07.059775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:07.059775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.059775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:07.059775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:07.059775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:07.059775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:07.059775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:07.059775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:07.060451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:07.060451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:07.060451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.060451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:07.060451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:07.060451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.060451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:07.060451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.060451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:07.060451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.060451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:07.060451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.062191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:07.062982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:07.062982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:07.062982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:07.062982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:07.062982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:07.062982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:07.063298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.063298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:07.063298 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.063298 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:07.063298 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:07.063298 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.063298 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:07.063298 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.063298 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:07.063298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.063298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:07.063298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:07.063298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:07.063298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.063298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:07.064333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:07.064333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.064333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:07.064333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:07.064333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:07.064333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.064333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:07.064333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:07.064333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:07.064333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:07.064333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:07.064333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:07.064333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:07.064333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:07.066013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.066013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:07.066013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:07.066013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.066013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:07.066013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.066013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:07.066013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.066013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:07.066013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:07.066013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:07.066013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:07.066013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:07.067170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:07.068956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:07.068956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:07.068956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:07.068956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:07.068956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:07.068956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.068956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:07.068956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:07.068956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.068956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:07.068956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.068956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:07.068956 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:07.068956 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:07.069266 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:07.069266 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:07.095695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:07.096430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.097613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:07.097613 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.097613 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:07.097613 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:07.097613 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.097613 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:07.097613 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.097613 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:07.098706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.093 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.098706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:07.098706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:07.098706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:07.098706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.098706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:07.098706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:07.098706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.098706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:07.098706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:07.098706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:07.098706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.099139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:07.099139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:07.099139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:07.099139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:07.099139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:07.099139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:07.099139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:07.099139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:07.099139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.099139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:07.099139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:07.099139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.099139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:07.099463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.099463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:07.099874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.099874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:07.099874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:07.099874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:07.099874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:07.099874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:07.099874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.099874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:07.099874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:07.099874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.099874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:07.099874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:07.099874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.099874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:07.099874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:07.099874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:07.100361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:07.100994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:07.100994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:07.100994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:11.867007 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:11.867686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:11.867686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:11.870706 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:11.870930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:11.871079 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:11.871248 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:11.871411 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:11.871587 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:11.871703 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:11.871807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:11.873820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.874379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:11.874379 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:11.874379 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:11.874379 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:11.874379 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:11.874379 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:11.874379 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.874379 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:11.875317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.875317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:11.875317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:11.875317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:11.875317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.875317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:11.875317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:11.875317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.875317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:11.875317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:11.875317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:11.875317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.875829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:11.875829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:11.875829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:11.875829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:11.875829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:11.875829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:11.875829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:11.875829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:11.875829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:11.875829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:11.875829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:11.875829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:11.875829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:11.876300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.876300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:11.876553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.876553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:11.876553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:11.876553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:11.876553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:11.876553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:11.876553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.876553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:11.876553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:11.876553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.876553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:11.876553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:11.876553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.876553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:11.876553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:11.876553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:11.877097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:11.877744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:11.877744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.877744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:11.902964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:11.904197 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:11.904197 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:11.906647 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:38:11.908256 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:11.908337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.909309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:11.909309 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:11.909309 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:11.909309 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:11.909309 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:11.909309 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:11.909309 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.909309 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:11.920302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.920302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:11.920302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:11.920302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:11.920302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.920302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:11.920302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:11.920302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:11.920302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.920302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:11.920302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:11.920302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:11.920302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:11.920302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:11.920302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:11.921286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:11.921286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:11.921286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:11.921286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:11.921286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:11.921286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:11.921286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:11.921286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.921286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:11.922838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.922838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:11.922838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:11.922838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:11.922838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.922838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:11.922838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:11.922838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:38:11.922838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:11.922838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:11.922838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.922838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:11.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:11.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:11.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:11.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:11.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:11.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:11.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:11.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:11.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:11.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:11.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:11.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:11.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:11.924318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:11.924318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:11.924318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.924318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:11.927108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.927108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:11.927108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:11.927108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:11.927108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:11.927108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:11.927108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.927108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:11.927108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:11.927108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.927108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:11.927108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:11.927108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.927108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:11.927108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:11.927108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:11.927726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:11.928401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:11.928401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:11.928401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:11.928401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:11.928401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:11.928401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.928401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:11.934687 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:11.935807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:11.935807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:11.935807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:11.936929 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:11.937367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.938310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:11.938310 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:11.938310 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:11.938310 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:11.938310 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:11.938310 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:11.938310 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.938310 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:11.939554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.939554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:11.939554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:11.939554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:11.939554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.939554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:11.939554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:11.939554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:11.939554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.939554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:11.939554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:11.939554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:11.939554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:11.939554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:11.939554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:11.939935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:11.939935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:11.939935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:11.939935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:11.939935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:11.939935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:11.939935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:11.939935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.939935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:11.940944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.940944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:11.940944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:11.940944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:11.940944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.940944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:11.940944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:11.940944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:11.940944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:11.940944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.940944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:11.941531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:11.941531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:11.941531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.941531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:11.941531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:11.941531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:11.941531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:11.941531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:11.941531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:11.941531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:11.941531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:11.941531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:11.941531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:11.941531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:11.942123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:11.942123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:11.942123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.942123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.946349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:11.947602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:11.948244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:11.948244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:11.948244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.948244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:11.973952 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:11.974651 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:11.974775 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:11.974920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:11.975457 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:11.975820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:11.976200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.976697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:11.976697 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:11.976697 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:11.976697 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:11.976697 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:11.976697 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:11.976697 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.976697 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:11.978509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.978509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:11.978509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:11.978509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:11.978509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.978509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:11.978509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:11.978509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.978509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:11.978509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:11.978509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:11.978509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.979053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:11.979053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:11.979053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:11.979053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:11.979053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:11.979053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:11.979053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:11.979053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:11.979053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:11.979053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:11.979053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:11.979053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:11.979053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:11.979346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.979346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:11.979986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.979986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:11.979986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:11.979986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:11.979986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:11.979986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:11.979986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.979986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:11.979986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:11.979986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.979986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:11.979986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:11.979986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.979986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:11.979986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:11.979986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:11.980783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:11.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:11.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:11.981607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:12.004438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:12.005963 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:12.007297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:12.008872 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:12.071792 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:12.072589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:12.072759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:12.072759 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:12.072759 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:12.072759 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:12.072759 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:12.072759 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:12.072759 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:12.072759 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:12.075348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.066 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:12.075348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:12.075348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:12.075348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:12.075348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:12.075348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:12.075348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:12.075348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:12.075348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:12.075348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:12.075348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:12.075348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:12.075879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:12.075879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:12.075879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:12.075879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:12.075879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:12.075879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:12.075879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:12.075879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:12.075879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:12.075879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:12.075879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:12.075879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:12.075879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:12.076140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:12.076140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:12.076621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:12.076621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:12.076621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:12.076621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:12.076621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:12.076621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:12.076621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:12.076621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:12.076621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:12.076621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:12.076621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:12.076621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:12.076621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:12.076621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:12.076621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:12.076621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:12.076928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:12.077269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:12.077269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:12.077269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:16.867501 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:16.872374 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:16.872374 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:16.872374 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:16.872374 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:16.872374 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:16.874644 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:16.874644 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:16.874644 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:16.874644 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:16.874644 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:16.876020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:16.876020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.876020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:16.876020 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:16.876020 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:16.876020 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:16.876020 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:16.876020 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:16.876020 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.876020 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:16.876020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.876020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:16.876020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:16.876020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:16.877589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.877589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:16.877589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:16.877589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.877589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:16.877589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:16.877589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:16.877589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.877589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:16.877589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:16.877589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:16.877589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:16.877589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:16.877589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:16.878660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:16.878660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:16.878660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:16.878660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:16.878660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:16.878660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:16.878660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:16.878660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.878660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:16.878660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.878660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:16.878660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:16.878999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:16.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:16.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:16.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:16.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:16.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:16.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:16.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:16.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:16.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:16.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:16.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:16.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:16.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.879412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:16.903566 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:16.903885 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:16.904000 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:16.904080 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:38:16.905114 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:16.905114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:16.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:16.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:16.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:16.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:16.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:16.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.905795 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:16.906495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.906495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:16.906495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:16.906495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:16.906495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.906495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:16.906495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:16.906495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:16.906495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.906495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:16.906495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:16.906495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:16.906495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:16.906495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:16.906495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:16.907937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:16.907937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:16.907937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:16.907937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:16.907937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:16.907937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:16.907937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:16.907937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.907937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:16.907937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.907937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:16.907937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:16.907937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:16.907937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.909313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:16.909313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:16.909313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:38:16.909313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:16.909313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:16.909313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.909313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:16.909313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:16.909313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:16.909313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.909313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:16.909313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:16.909313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:16.910070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:16.910070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:16.910070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:16.910070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:16.910070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:16.910070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:16.910070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:16.910070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:16.910070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:16.910070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:16.910070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.910070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:16.910309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.910309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:16.910309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:16.910309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:16.910309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:16.910309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:16.910309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.910309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:16.910309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:16.910309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.910309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:16.910309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:16.910309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.910309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:16.910309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:16.910309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:16.910656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:16.911096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:16.911096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:16.911096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:16.911096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:16.911096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:16.911096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.911096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:16.935779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:16.937910 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:16.938025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:16.938180 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:16.939323 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:16.939323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.939501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:16.939501 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:16.939501 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:16.939501 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:16.939501 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:16.939501 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:16.939501 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.939501 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:16.940362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.940362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:16.940362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:16.940362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:16.940362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.940362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:16.940362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:16.940362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:16.940362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.940362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:16.940362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:16.940362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:16.940362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:16.940362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:16.940362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:16.941404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:16.941404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:16.941404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:16.941404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:16.941404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:16.941404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:16.941404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:16.941404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.941404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:16.943437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.943437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:16.943437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:16.943437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:16.943437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.943437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:16.943437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:16.943437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:16.943437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:16.943437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.943437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:16.944417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:16.944417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:16.944417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.944417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:16.944417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:16.944417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:16.944417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:16.944417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:16.944417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:16.944417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:16.944417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:16.944417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:16.944417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:16.944417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:16.944802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:16.944802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:16.944802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.944802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.946320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:16.947067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:16.947471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:16.947471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:16.947471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.947471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:16.974796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:16.975287 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:16.975287 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:16.975387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:16.976676 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:16.976676 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:16.976676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.977413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:16.977413 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:16.977413 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:16.977413 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:16.977413 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:16.977413 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:16.977413 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.977413 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:16.978316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.978316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:16.978316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:16.978316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:16.978316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.978316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:16.978316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:16.978316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.978316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:16.978316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:16.978316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:16.978316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.978579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:16.978579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:16.978579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:16.978579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:16.978579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:16.978579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:16.978579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:16.978579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:16.978579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:16.978579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:16.978579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:16.978579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:16.978579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:16.978785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.978785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:16.979085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.979085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:16.979085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:16.979085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:16.979085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:16.979085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:16.979085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.979085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:16.979085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:16.979085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.979085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:16.979085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:16.979085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.979085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:16.979085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:16.979085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:16.979670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:16.979966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:16.979966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:16.979966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:17.005104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:17.005533 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:17.005795 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:17.010373 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:17.061454 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:17.068809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:17.068809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:17.068809 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:17.068809 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:17.068809 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:17.068809 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:17.068809 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:17.068809 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:17.068809 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:17.069189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.063 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:17.069189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:17.069189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:17.069189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:17.069189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:17.069189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:17.069189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:17.069189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:17.069189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:17.069189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:17.069189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:17.069189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:17.069525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:17.069525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:17.069525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:17.069525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:17.069525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:17.069525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:17.069525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:17.069525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:17.069525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:17.069525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:17.069525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:17.069525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:17.069525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:17.069902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:17.069902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:17.069962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:17.069962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:17.069962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:17.069962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:17.069962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:17.069962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:17.069962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:17.069962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:17.069962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:17.069962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:17.069962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:17.069962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:17.069962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:17.069962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:17.069962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:17.069962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:17.070390 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:17.070932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:17.070932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:17.070932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:21.868281 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:21.871118 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:21.871118 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:21.871118 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:21.871118 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:21.871118 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:21.872122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:21.872122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:21.872122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:21.872122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:21.872122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:21.874664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:21.874664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:21.874664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.874664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:21.874664 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:21.874664 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:21.874664 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:21.874664 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:21.874664 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:21.874664 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.874664 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:21.874664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.875674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:21.875674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:21.875674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:21.875674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.875674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:21.875674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:21.875674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.875674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:21.875674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:21.875674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:21.875674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.875674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:21.875674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:21.875674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:21.877884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:21.877884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:21.877884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:21.877884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:21.877884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:21.877884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:21.877884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:21.877884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:21.877884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:21.877884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:21.877884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.877884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:21.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:21.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:21.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:21.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:21.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:21.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:21.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:21.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:21.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:21.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:21.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:21.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:21.878859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:21.879680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:21.879680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.879680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:21.904022 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:21.904388 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:21.904640 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:21.904817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:38:21.905200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:21.905386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.905908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:21.905908 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:21.905908 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:21.905908 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:21.905908 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:21.905908 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:21.905908 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.905908 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:21.906559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.906559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:21.906559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:21.906559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:21.906559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.906559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:21.906559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:21.906559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:21.906559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.906559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:21.906559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:21.906559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:21.906559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:21.906559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:21.906559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:21.907704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:21.907704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:21.907704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:21.907704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:21.907704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:21.907704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:21.907704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:21.907704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.907704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:21.907704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.907704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:21.907704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:21.907704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:21.907704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.907975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:21.907975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:21.907975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:38:21.907975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:21.907975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:21.907975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.907975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:21.907975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:21.907975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:21.907975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.907975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:21.907975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:21.907975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:21.908175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:21.908175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:21.908175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:21.908175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:21.908175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:21.908175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:21.908175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:21.908175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:21.908175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:21.908175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:21.908175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.908175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:21.908383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.908383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:21.908383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:21.908383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:21.908383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:21.908383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:21.908383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.908383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:21.908383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:21.908383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.908383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:21.908383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:21.908383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.908383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:21.908383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:21.908383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:21.908659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:21.909059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:21.909059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:21.909059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:21.909059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:21.909059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:21.909059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.909059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:21.938153 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:21.938677 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:21.938845 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:21.938907 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:21.939345 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:21.939557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.940268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:21.940268 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:21.940268 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:21.940268 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:21.940268 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:21.940268 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:21.940268 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.940268 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:21.941028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.941028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:21.941028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:21.941028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:21.941028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.941028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:21.941028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:21.941028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:21.941028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.941028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:21.941028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:21.941028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:21.941028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:21.941028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:21.941028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:21.941354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:21.941354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:21.941354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:21.941354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:21.941354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:21.941354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:21.941354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:21.941354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.941354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:21.941354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.941354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:21.941354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:21.941354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:21.941354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.943094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:21.943094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:21.943094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:21.943094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:21.943094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.943094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:21.943094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:21.943094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:21.943094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.943094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:21.943094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:21.943094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:21.943094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:21.943961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:21.943961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:21.943961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:21.943961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:21.943961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:21.943961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:21.943961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:21.943961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:21.943961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:21.943961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.943961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:21.943961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:21.944276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.944878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:21.976822 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:21.977713 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:21.977713 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:21.978234 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:21.978387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:21.978730 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:21.979346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.979863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:21.979863 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:21.979863 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:21.979863 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:21.979863 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:21.979863 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:21.979863 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.979863 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:21.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:21.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:21.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:21.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:21.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:21.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:21.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:21.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:21.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.982979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:21.982979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:21.982979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:21.982979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:21.982979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:21.982979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:21.982979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:21.982979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:21.982979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:21.982979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:21.982979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:21.982979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:21.982979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:21.983254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.983254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:21.983254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.983254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:21.983254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:21.983254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:21.983254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:21.983254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:21.983254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.983254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:21.983254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:21.983254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.983254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:21.983254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:21.983254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:21.983759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:21.984381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:21.984381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:21.984381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:21.984381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:21.984381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:21.984381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:22.005636 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:22.006050 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:22.006239 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:22.007555 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:22.009214 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=240 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:38:22.059625 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:22.059868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:22.060331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:22.060331 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:22.060331 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:22.060331 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:22.060331 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:22.060331 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:22.060331 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:22.060331 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:22.061004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.054 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:22.061004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:22.061004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:22.061004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:22.061004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:22.061004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:22.061004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:22.061004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:22.061004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:22.061004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:22.061004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:22.061004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:22.061280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:22.061280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:22.061280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:22.061280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:22.061280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:22.061280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:22.061280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:22.061280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:22.061280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:22.061280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:22.061280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:22.061280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:22.061280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:22.061524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:22.061524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:22.061819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:22.061819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:22.061819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:22.061819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:22.061819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:22.061819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:22.061819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:22.061819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:22.061819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:22.061819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:22.061819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:22.061819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:22.061819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:22.061819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:22.061819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:22.061819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:22.062087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:22.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:22.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:22.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:26.868841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:26.869457 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:26.869457 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:26.870497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:26.870825 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:26.871016 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:26.871195 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:26.871362 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:26.871578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:26.871724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:26.871888 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:26.872046 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:26.872213 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:26.873055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.873109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:26.873109 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:26.873109 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:26.873109 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:26.873109 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:26.873109 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:26.873109 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.873109 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:26.873904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.873904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:26.873904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:26.873904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:26.873904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.873904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:26.873904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:26.873904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.873904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:26.873904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:26.873904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:26.873904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.874205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:26.874205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:26.874205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:26.874205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:26.874205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:26.874205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:26.874205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:26.874205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:26.874205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:26.874205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:26.874205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:26.874205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:26.874205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:26.874556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.874556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:26.874909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.874909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:26.874909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:26.874909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:26.874909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:26.874909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:26.874909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.874909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:26.874909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:26.874909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.874909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:26.874909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:26.874909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.874909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:26.874909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:26.874909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:26.875297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:26.875716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:26.875716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.875716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:26.904495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:26.904855 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:26.904945 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:26.905080 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:38:26.905422 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:26.905649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.906152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:26.906152 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:26.906152 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:26.906152 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:26.906152 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:26.906152 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:26.906152 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.906152 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:26.906755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.906755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:26.906755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:26.906755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:26.906755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.906755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:26.906755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:26.906755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:26.906755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.906755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:26.906755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:26.906755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:26.906755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:26.906755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:26.906755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:26.907092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:26.907092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:26.907092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:26.907092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:26.907092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:26.907092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:26.907092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:26.907092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.907092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:26.907513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.907513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:26.907513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:26.907513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:26.907513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.907513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:26.907513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:26.907513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:38:26.907513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:26.907513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:26.907513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.907513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:26.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:26.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:26.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:26.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:26.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:26.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:26.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:26.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:26.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:26.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:26.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:26.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:26.907811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:26.908107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:26.908107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:26.908107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.908107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:26.908479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.908479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:26.908479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:26.908479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:26.908479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:26.908479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:26.908479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.908479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:26.908479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:26.908479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.908479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:26.908479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:26.908479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.908479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:26.908479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:26.908479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:26.909230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:26.909580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:26.909580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:26.909580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:26.909580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:26.909580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:26.909580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.909580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:26.938632 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:26.938865 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:26.938980 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:26.939110 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:26.939510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:26.939816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.940269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:26.940269 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:26.940269 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:26.940269 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:26.940269 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:26.940269 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:26.940269 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.940269 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:26.940911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.940911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:26.940911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:26.940911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:26.940911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.940911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:26.940911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:26.940911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:26.940911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.940911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:26.940911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:26.940911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:26.940911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:26.940911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:26.940911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:26.941216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:26.941216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:26.941216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:26.941216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:26.941216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:26.941216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:26.941216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:26.941216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.941216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:26.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:26.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:26.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:26.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:26.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:26.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:26.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:26.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:26.941965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:26.941965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:26.941965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.941965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:26.941965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:26.941965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:26.941965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:26.941965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:26.941965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:26.941965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:26.941965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:26.941965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:26.941965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:26.941965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:26.942184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:26.942184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:26.942184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.942184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.942314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:26.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:26.942994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:26.942994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:26.942994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.942994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:26.977312 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:26.977983 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:26.977983 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:26.977983 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:26.980892 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:26.981700 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:26.982002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:26.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:26.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:26.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:26.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:26.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:26.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.982588 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:26.983730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.983730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:26.983730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:26.983730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:26.983730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.983730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:26.983730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:26.983730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.983730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:26.983730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:26.983730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:26.983730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.984118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:26.984118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:26.984118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:26.984118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:26.984118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:26.984118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:26.984118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:26.984118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:26.984118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:26.984118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:26.984118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:26.984118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:26.984118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:26.984432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.984432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:26.984652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.984652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:26.984652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:26.984652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:26.984652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:26.984652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:26.984652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.984652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:26.984652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:26.984652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.984652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:26.984652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:26.984652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.984652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:26.984652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:26.984652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:26.985055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:26.985824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:26.985824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:26.985824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:27.006065 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:27.006370 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:27.006679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:27.007062 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:27.044614 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:27.072624 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.072928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.073596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:27.073596 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.073596 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:27.073596 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:27.073596 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.073596 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:27.073596 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.073596 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:27.074392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.067 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.074392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:27.074392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:27.074392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:27.074392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:27.074392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:27.074392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:27.074392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.074392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.074392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:27.074392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:27.074392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.074843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.074843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:27.074843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:27.074843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:27.074843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:27.074843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:27.074843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:27.074843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:27.074843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.074843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:27.074843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:27.074843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.074843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:27.075228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.075228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:27.075228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.075228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:27.075228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:27.075228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:27.075228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:27.075228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:27.075228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:27.075228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:27.075228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:27.075228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:27.075228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:27.075228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:27.075228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:27.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.076248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:27.076248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:27.076248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.076248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:27.076248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.076248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:27.076248 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:27.076248 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:27.082316 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 01 11:38:27.082706 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Number of inconsistencies found at create/update: networks=8, security_groups=2, routers=5, router_ports=5 {{(pid=89427) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 01 11:38:27.082706 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=14 {{(pid=89427) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 01 11:38:27.082706 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 00ec979f-ff41-4623-9a26-50e70b52e7e9 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:27.145096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-00ec979f-ff41-4623-9a26-50e70b52e7e9, resource={'id': '00ec979f-ff41-4623-9a26-50e70b52e7e9', 'name': 'tempest-TestEncryptedCinderVolumes-1927074611-network', 'tenant_id': 'b66197cfa79c4f6b893b9fdeed880160', 'project_id': 'b66197cfa79c4f6b893b9fdeed880160', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['c0944124-235a-493b-9743-59a04ed5151d'], 'standard_attr_id': 1177, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:36:59Z', 'updated_at': '2026-06-01T11:37:00Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 11608}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.145449 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-00ec979f-ff41-4623-9a26-50e70b52e7e9, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestEncryptedCinderVolumes-1927074611-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.145869 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-da620088-df13-4203-8ab8-5646f165407b, external_ids_update=None, columns={'tag': 11608}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.146133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.147124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:27.147124 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.147124 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:27.147124 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:27.147124 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.147124 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:27.147124 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.147124 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:27.148801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 00ec979f-ff41-4623-9a26-50e70b52e7e9 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.148801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:27.148801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:27.148801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:27.148801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:27.148801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:27.148801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:38:27.148801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:27.148801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.148801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.148801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:27.149752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:27.149752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.149752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.149752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:27.149752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:27.149752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:27.149752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:27.149752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:27.149752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:27.149752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:27.149752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.149752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:27.149752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:27.149752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.150103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:27.150103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.150103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:27.150103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 0d4134e5-3b3a-49d4-a466-c1b6f993b0ab (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:27.182082 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-0d4134e5-3b3a-49d4-a466-c1b6f993b0ab, resource={'id': '0d4134e5-3b3a-49d4-a466-c1b6f993b0ab', 'name': 'private', 'tenant_id': '43b6be50ae544caeb6a73eced65fb6a7', 'project_id': '43b6be50ae544caeb6a73eced65fb6a7', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['16527f37-b4ae-44e5-9382-abb5e59d1cad', 'a33d59fc-0fd4-45c5-9fff-82704a1922a6'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:23:10Z', 'updated_at': '2026-06-01T11:23:13Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 48058}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.183547 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-0d4134e5-3b3a-49d4-a466-c1b6f993b0ab, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.183967 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-bedd3a6a-831e-4f6a-adf7-0842bd968bac, external_ids_update=None, columns={'tag': 48058}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.184109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.185780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:27.185780 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.185780 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:27.185780 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:27.185780 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.185780 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:27.185780 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.185780 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:27.187176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 0d4134e5-3b3a-49d4-a466-c1b6f993b0ab (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.187176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:27.187176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:27.187176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:27.187176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:27.187176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:27.187176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:38:27.187176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:27.187176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.187176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.187176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:27.187409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:27.187409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.187409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.187409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:27.187409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:27.187409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:27.187409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:27.187409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:27.187409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:27.187409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:27.187409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.187409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:27.187409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:27.187409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.187750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:27.187750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.187750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:27.187750 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 19da64d5-62e3-4b38-84f1-7d0971806ce9 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:27.232115 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-19da64d5-62e3-4b38-84f1-7d0971806ce9, resource={'id': '19da64d5-62e3-4b38-84f1-7d0971806ce9', 'name': 'shared', 'tenant_id': '43b6be50ae544caeb6a73eced65fb6a7', 'project_id': '43b6be50ae544caeb6a73eced65fb6a7', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['c855657f-8ba2-422e-88fd-c5a71ec03ed7'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:24:23Z', 'updated_at': '2026-06-01T11:24:23Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 19987}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.232429 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-19da64d5-62e3-4b38-84f1-7d0971806ce9, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.232870 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-cb9dd134-19dd-491b-aafb-e4edb81fcf60, external_ids_update=None, columns={'tag': 19987}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.233080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.235014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:27.235014 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.235014 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:27.235014 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:27.235014 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.235014 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:27.235014 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.235014 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:27.236117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 19da64d5-62e3-4b38-84f1-7d0971806ce9 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.236117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:27.236117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:27.236117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:27.236117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:27.236117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:27.236117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:38:27.236117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:27.236117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.236117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.236117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:27.236693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:27.236693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.236693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.236693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:27.236693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:27.236693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:27.236693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:27.236693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:27.236693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:27.236693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:27.236693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.236693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:27.236693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:27.236693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.237051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:27.237051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.237051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:27.237051 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 3492f8e6-3ea9-42e6-986a-8ca849074669 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:27.283216 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-3492f8e6-3ea9-42e6-986a-8ca849074669, resource={'id': '3492f8e6-3ea9-42e6-986a-8ca849074669', 'name': 'tempest-TestEncryptedCinderVolumes-1394906684-network', 'tenant_id': 'd77c4e1763dc48c7a35074bdb31f0f9f', 'project_id': 'd77c4e1763dc48c7a35074bdb31f0f9f', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['ca56285f-442a-4f1a-8a7e-1b4f68f085a5'], 'standard_attr_id': 867, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:34:10Z', 'updated_at': '2026-06-01T11:34:10Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 23558}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.283424 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-3492f8e6-3ea9-42e6-986a-8ca849074669, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestEncryptedCinderVolumes-1394906684-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.283834 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-94d05f2e-959d-48f3-a414-21a304e82be7, external_ids_update=None, columns={'tag': 23558}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.284032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.284616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:27.284616 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.284616 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:27.284616 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:27.284616 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.284616 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:27.284616 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.284616 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:27.286782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 3492f8e6-3ea9-42e6-986a-8ca849074669 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.286782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:27.286782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:27.286782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:27.286782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:27.286782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:27.286782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:38:27.286782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:27.286782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.286782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.286782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:27.287110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:27.287110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.287110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.287110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:27.287110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:27.287110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:27.287110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:27.287110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:27.287110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:27.287110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:27.287110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.287110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:27.287110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:27.287110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.287555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:27.287555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.287555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:27.287555 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 43580e48-0b8f-4944-8cef-00120baaab9a (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:27.344484 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-43580e48-0b8f-4944-8cef-00120baaab9a, resource={'id': '43580e48-0b8f-4944-8cef-00120baaab9a', 'name': 'tempest-TestEncryptedCinderVolumes-1119314671-network', 'tenant_id': '027edaf7ca764c88afca90c7bbbc1437', 'project_id': '027edaf7ca764c88afca90c7bbbc1437', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['b3dd426c-171d-4a0b-970c-69ad3ce25b9c'], 'standard_attr_id': 1165, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:36:54Z', 'updated_at': '2026-06-01T11:36:55Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 25178}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.344484 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-43580e48-0b8f-4944-8cef-00120baaab9a, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestEncryptedCinderVolumes-1119314671-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.344735 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-b4461f97-f192-4056-823d-cf84add74387, external_ids_update=None, columns={'tag': 25178}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.344735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.344735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:27.344735 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.344735 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:27.344735 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:27.344735 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.344735 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:27.344735 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.344735 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:27.345712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 43580e48-0b8f-4944-8cef-00120baaab9a (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.345712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:27.345712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:27.345712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:27.345712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:27.345712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:27.345712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:38:27.345712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:27.345712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.345712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.345712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:27.348742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:27.348742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.348742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.348742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:27.348742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:27.348742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:27.348742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:27.348742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:27.348742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:27.348742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:27.348742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.348742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:27.348742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:27.348742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.349046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:27.349046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.349046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:27.349046 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 45586b37-8c68-4975-bc3b-8e3230ced693 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:27.383975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-45586b37-8c68-4975-bc3b-8e3230ced693, resource={'id': '45586b37-8c68-4975-bc3b-8e3230ced693', 'name': 'tempest-TestEncryptedCinderVolumes-1366858945-network', 'tenant_id': '3fb7e4fd9c544b28a60ab0a49b43c46b', 'project_id': '3fb7e4fd9c544b28a60ab0a49b43c46b', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['ffbd1d55-ab6d-4849-8003-322763ee8a09'], 'standard_attr_id': 854, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:34:02Z', 'updated_at': '2026-06-01T11:34:03Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 1331}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.384260 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-45586b37-8c68-4975-bc3b-8e3230ced693, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestEncryptedCinderVolumes-1366858945-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.384551 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-64ca6c6b-97ac-45ef-842b-704b4e3b7689, external_ids_update=None, columns={'tag': 1331}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.384667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.385272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:27.385272 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.385272 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:27.385272 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:27.385272 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.385272 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:27.385272 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.385272 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:27.386381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 45586b37-8c68-4975-bc3b-8e3230ced693 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.386381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:27.386381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:27.386381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:27.386381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:27.386381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:27.386381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:38:27.386381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:27.386381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.386381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.386381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:27.387013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:27.387013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.387013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.387013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:27.387013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:27.387013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:27.387013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:27.387013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:27.387013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:27.387013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:27.387013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.387013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:27.387013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:27.387013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.387346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:27.387346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.387346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:27.387346 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 863342db-68a8-4b21-9592-b676ff2c101b (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:27.420375 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-863342db-68a8-4b21-9592-b676ff2c101b, resource={'id': '863342db-68a8-4b21-9592-b676ff2c101b', 'name': 'tempest-AttachVolumeTestJSON-1275615359-network', 'tenant_id': '85b2cea923b94a958d3f76298db530a3', 'project_id': '85b2cea923b94a958d3f76298db530a3', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['7a096ad4-f3eb-436d-9a7d-c0d686cff9f8'], 'standard_attr_id': 962, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:35:08Z', 'updated_at': '2026-06-01T11:35:09Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 16189}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.420628 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-863342db-68a8-4b21-9592-b676ff2c101b, col_values=(('external_ids', {'neutron:network_name': 'tempest-AttachVolumeTestJSON-1275615359-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.420879 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-7950dc51-5f57-4bd0-a5bc-1c4cdc9cbf80, external_ids_update=None, columns={'tag': 16189}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.420961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.423869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:27.423869 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.423869 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:27.423869 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:27.423869 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.423869 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:27.423869 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.423869 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:27.425764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 863342db-68a8-4b21-9592-b676ff2c101b (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.425764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:27.425764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:27.425764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:27.425764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:27.425764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:27.425764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:38:27.425764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:27.425764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.425764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.425764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:27.426152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:27.426152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.426152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.426152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:27.426152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:27.426152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:27.426152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:27.426152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:27.426152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:27.426152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:27.426152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.426152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:27.426152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:27.426152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.426621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:27.426621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.426621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:27.426621 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource c6c73189-2fc5-412b-9ed5-038efd7341bc (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:27.457407 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, resource={'id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'name': 'public', 'tenant_id': '43b6be50ae544caeb6a73eced65fb6a7', 'project_id': '43b6be50ae544caeb6a73eced65fb6a7', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['21571d58-7621-44df-bdae-bad06600904d', '2bd944db-99d5-4573-8fc8-07e7942a1537'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-01T11:23:15Z', 'updated_at': '2026-06-01T11:23:22Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.457681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.457931 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-4ecb0f57-2a36-41bc-88d1-5cf4f555b097, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.458087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.458552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:27.458552 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.458552 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:27.458552 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:27.458552 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.458552 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:27.458552 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.458552 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:27.459629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource c6c73189-2fc5-412b-9ed5-038efd7341bc (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.459629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:27.459629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:27.459629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:27.459629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:27.459629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:27.459629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:38:27.459629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:27.459629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.459629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.459629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:27.459846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:27.459846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.459846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.459846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:27.459846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:27.459846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:27.459846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:27.459846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:27.459846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:27.459846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:27.459846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.459846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:27.459846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:27.459846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.460057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:27.460057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.460057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:27.460057 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 5b42f08a-4696-4372-90f8-e99439e6b1be (type: security_groups) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:27.492835 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Successfully bumped revision number for resource 5b42f08a-4696-4372-90f8-e99439e6b1be (type: security_groups) to 3 Jun 01 11:38:27.493037 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource a9280724-8f33-4c5a-b2a1-bfb3f319a6ff (type: security_groups) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:27.526291 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Successfully bumped revision number for resource a9280724-8f33-4c5a-b2a1-bfb3f319a6ff (type: security_groups) to 3 Jun 01 11:38:27.526452 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 065f5984-bcd4-48bd-b8cb-3119f0d33351 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:27.609684 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, resource={'id': '065f5984-bcd4-48bd-b8cb-3119f0d33351', 'name': 'tempest-AttachVolumeTestJSON-1275615359-router', 'project_id': '85b2cea923b94a958d3f76298db530a3', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.173'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::64'}], 'enable_snat': True}, 'gw_port_id': 'b76e8c43-e6a5-4e0e-ae38-a143937e6670', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.173', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::64', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:35:10Z', 'updated_at': '2026-06-01T11:35:12Z', 'revision_number': 3, 'tenant_id': '85b2cea923b94a958d3f76298db530a3'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.609893 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, columns={'external_ids': {'neutron:router_name': 'tempest-AttachVolumeTestJSON-1275615359-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.610121 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "b76e8c43-e6a5-4e0e-ae38-a143937e6670"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.610260 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "b76e8c43-e6a5-4e0e-ae38-a143937e6670"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.610366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.611097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:27.611097 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.611097 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:27.611097 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:27.611097 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.611097 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:27.611097 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.611097 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:27.611317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router 065f5984-bcd4-48bd-b8cb-3119f0d33351. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.613002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 065f5984-bcd4-48bd-b8cb-3119f0d33351 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.613002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:27.613002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:27.613002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:27.613002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:27.613002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:27.613002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:38:27.613002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:38:27.613002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:38:27.613002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:38:27.613002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:38:27.613280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:38:27.613280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:38:27.613280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:27.613280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.613280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.613280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:27.613280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:27.613280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.613280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.613280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:27.613280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:27.613280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:27.613280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:27.613280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:27.613565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:27.613565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:27.613565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.613565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:27.613565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:27.613565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.613565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:27.613565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.613565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:27.613565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 0887475e-f238-4c9c-ab00-0385551e6fa3 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:27.692755 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, resource={'id': '0887475e-f238-4c9c-ab00-0385551e6fa3', 'name': 'tempest-TestEncryptedCinderVolumes-1394906684-router', 'project_id': 'd77c4e1763dc48c7a35074bdb31f0f9f', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.211'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::221'}], 'enable_snat': True}, 'gw_port_id': '05e73b9a-1af7-4f42-b8ae-0d3d7873b442', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.211', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::221', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:34:11Z', 'updated_at': '2026-06-01T11:34:14Z', 'revision_number': 3, 'tenant_id': 'd77c4e1763dc48c7a35074bdb31f0f9f'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.692989 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, columns={'external_ids': {'neutron:router_name': 'tempest-TestEncryptedCinderVolumes-1394906684-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.693169 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "05e73b9a-1af7-4f42-b8ae-0d3d7873b442"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.693288 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "05e73b9a-1af7-4f42-b8ae-0d3d7873b442"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.693382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.694075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:27.694075 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.694075 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:27.694075 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:27.694075 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.694075 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:27.694075 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.694075 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:27.694299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router 0887475e-f238-4c9c-ab00-0385551e6fa3. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.695234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 0887475e-f238-4c9c-ab00-0385551e6fa3 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.695234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:27.695234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:27.695234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:27.695234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:27.695234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:27.695234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:38:27.695234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:38:27.695234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:38:27.695234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:38:27.695234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:38:27.695585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:38:27.695585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:38:27.695585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:27.695585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.695585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.695585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:27.695585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:27.695585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.695585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.695585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:27.695585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:27.695585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:27.695585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:27.695585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:27.695953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:27.695953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:27.695953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.695953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:27.695953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:27.695953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.695953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:27.695953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.695953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:27.695953 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 7f17481f-41b4-4cca-9c5f-2f20e531ae95 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:27.775751 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, resource={'id': '7f17481f-41b4-4cca-9c5f-2f20e531ae95', 'name': 'tempest-TestEncryptedCinderVolumes-1927074611-router', 'project_id': 'b66197cfa79c4f6b893b9fdeed880160', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.34'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::23'}], 'enable_snat': True}, 'gw_port_id': '91ac5e2c-0c45-47b0-a854-6f34ff4755d1', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.34', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::23', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:37:01Z', 'updated_at': '2026-06-01T11:37:03Z', 'revision_number': 3, 'tenant_id': 'b66197cfa79c4f6b893b9fdeed880160'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.775927 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, columns={'external_ids': {'neutron:router_name': 'tempest-TestEncryptedCinderVolumes-1927074611-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.775927 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "91ac5e2c-0c45-47b0-a854-6f34ff4755d1"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.775927 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "91ac5e2c-0c45-47b0-a854-6f34ff4755d1"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.775927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.776052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:27.776052 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.776052 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:27.776052 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:27.776052 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.776052 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:27.776052 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.776052 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:27.776202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router 7f17481f-41b4-4cca-9c5f-2f20e531ae95. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.777247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 7f17481f-41b4-4cca-9c5f-2f20e531ae95 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.777247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:27.777247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:27.777247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:27.777247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:27.777247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:27.777247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:38:27.777247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:38:27.777247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:38:27.777247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:38:27.777247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:38:27.777462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:38:27.777462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:38:27.777462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:27.777462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.777462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.777462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:27.777462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:27.777462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.777462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.777462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:27.777462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:27.777462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:27.777462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:27.777462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:27.777770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:27.777770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:27.777770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.777770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:27.777770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:27.777770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.777770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:27.777770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.777770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:27.777770 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource d032402b-d9bd-4edb-868c-286dafc01a55 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:27.853921 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, resource={'id': 'd032402b-d9bd-4edb-868c-286dafc01a55', 'name': 'tempest-TestEncryptedCinderVolumes-1119314671-router', 'project_id': '027edaf7ca764c88afca90c7bbbc1437', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.7'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::b9'}], 'enable_snat': True}, 'gw_port_id': '51ee1662-ddb6-47da-a1b7-1e9ce15ff013', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.7', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::b9', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:36:55Z', 'updated_at': '2026-06-01T11:36:57Z', 'revision_number': 3, 'tenant_id': '027edaf7ca764c88afca90c7bbbc1437'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, columns={'external_ids': {'neutron:router_name': 'tempest-TestEncryptedCinderVolumes-1119314671-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "51ee1662-ddb6-47da-a1b7-1e9ce15ff013"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "51ee1662-ddb6-47da-a1b7-1e9ce15ff013"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.855160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.855414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:27.855414 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.855414 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:27.855414 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:27.855414 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.855414 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:27.855414 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.855414 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:27.855653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router d032402b-d9bd-4edb-868c-286dafc01a55. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.856760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource d032402b-d9bd-4edb-868c-286dafc01a55 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.856760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:27.856760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:27.856760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:27.856760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:27.856760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:27.856760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:38:27.856760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:38:27.856760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:38:27.856760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:38:27.856760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:38:27.857076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:38:27.857076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:38:27.857076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:27.857076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.857076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.857076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:27.857076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:27.857076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.857076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.857076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:27.857076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:27.857076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:27.857076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:27.857076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:27.857327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:27.857327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:27.857327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.857327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:27.857327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:27.857327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.857327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:27.857327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.857327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:27.857327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource d1c522ad-0f59-4939-9391-c3533f4cb09e (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:27.933358 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, resource={'id': 'd1c522ad-0f59-4939-9391-c3533f4cb09e', 'name': 'tempest-TestEncryptedCinderVolumes-1366858945-router', 'project_id': '3fb7e4fd9c544b28a60ab0a49b43c46b', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.221'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::189'}], 'enable_snat': True}, 'gw_port_id': '64c1bc42-615e-4bd7-b6c0-b6e62f628072', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.221', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::189', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:34:04Z', 'updated_at': '2026-06-01T11:34:07Z', 'revision_number': 3, 'tenant_id': '3fb7e4fd9c544b28a60ab0a49b43c46b'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.933909 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, columns={'external_ids': {'neutron:router_name': 'tempest-TestEncryptedCinderVolumes-1366858945-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.933909 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "64c1bc42-615e-4bd7-b6c0-b6e62f628072"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.933909 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "64c1bc42-615e-4bd7-b6c0-b6e62f628072"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:27.933909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.933909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:27.933909 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.933909 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:27.933909 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:27.934095 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.934095 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:27.934095 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.934095 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:27.934095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router d1c522ad-0f59-4939-9391-c3533f4cb09e. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.934660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource d1c522ad-0f59-4939-9391-c3533f4cb09e (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.934660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:27.934660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:27.934660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:27.934660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:27.934660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:27.934660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:38:27.934660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:38:27.934660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:38:27.934660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:38:27.934660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:38:27.934976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:38:27.934976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:38:27.934976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:27.934976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.934976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.934976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:27.934976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:27.934976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:27.934976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:27.934976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:27.934976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:27.934976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:27.934976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:27.934976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:27.935192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:27.935192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:27.935192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:27.935192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:27.935192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:27.935192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:27.935192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:27.935192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:27.935192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:27.935192 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 05e73b9a-1af7-4f42-b8ae-0d3d7873b442 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:28.046106 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-05e73b9a-1af7-4f42-b8ae-0d3d7873b442, resource={'id': '05e73b9a-1af7-4f42-b8ae-0d3d7873b442', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:62:c5:5c', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '0887475e-f238-4c9c-ab00-0385551e6fa3', 'device_owner': 'network:router_gateway', 'standard_attr_id': 872, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.211'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::221'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:34:11Z', 'updated_at': '2026-06-01T11:34:14Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:28.049300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-05e73b9a-1af7-4f42-b8ae-0d3d7873b442, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-0887475e-f238-4c9c-ab00-0385551e6fa3'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::221/64', '172.24.5.211/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:28.049300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=05e73b9a-1af7-4f42-b8ae-0d3d7873b442, lrouter_port=lrp-05e73b9a-1af7-4f42-b8ae-0d3d7873b442, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:28.049300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.049605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:28.049605 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:28.049605 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:28.049605 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:28.049605 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:28.049605 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:28.049605 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.049605 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:28.051026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 05e73b9a-1af7-4f42-b8ae-0d3d7873b442 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.051026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:28.051026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:28.051026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:28.051026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:28.051026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:28.051026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:38:28.051026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:28.051026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:28.051026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:28.051026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:28.051267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:28.051267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:28.051267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:28.051267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:28.051267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:28.051267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:28.051267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:28.051267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:28.051267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:28.051267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:28.051267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:28.051267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:28.051267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:28.051267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:28.051500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:28.051500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.051500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:28.052040 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 51ee1662-ddb6-47da-a1b7-1e9ce15ff013 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:28.182869 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-51ee1662-ddb6-47da-a1b7-1e9ce15ff013, resource={'id': '51ee1662-ddb6-47da-a1b7-1e9ce15ff013', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:19:ae:3a', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'd032402b-d9bd-4edb-868c-286dafc01a55', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1170, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.7'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::b9'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:36:55Z', 'updated_at': '2026-06-01T11:36:58Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:28.184368 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-51ee1662-ddb6-47da-a1b7-1e9ce15ff013, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-d032402b-d9bd-4edb-868c-286dafc01a55'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::b9/64', '172.24.5.7/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:28.184975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=51ee1662-ddb6-47da-a1b7-1e9ce15ff013, lrouter_port=lrp-51ee1662-ddb6-47da-a1b7-1e9ce15ff013, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:28.185724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.186674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:28.186674 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:28.186674 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:28.186674 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:28.186674 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:28.186674 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:28.186674 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.186674 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:28.187579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 51ee1662-ddb6-47da-a1b7-1e9ce15ff013 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.187579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:28.187579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:28.187579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:28.187579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:28.187579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:28.187579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:38:28.187579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:28.187579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:28.187579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:28.187579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:28.187828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:28.187828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:28.187828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:28.187828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:28.187828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:28.187828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:28.187828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:28.187828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:28.187828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:28.187828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:28.187828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:28.187828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:28.187828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:28.187828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:28.188048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:28.188048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.188048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:28.188048 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 64c1bc42-615e-4bd7-b6c0-b6e62f628072 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:28.317561 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-64c1bc42-615e-4bd7-b6c0-b6e62f628072, resource={'id': '64c1bc42-615e-4bd7-b6c0-b6e62f628072', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:ec:e7:da', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'd1c522ad-0f59-4939-9391-c3533f4cb09e', 'device_owner': 'network:router_gateway', 'standard_attr_id': 860, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.221'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::189'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:34:04Z', 'updated_at': '2026-06-01T11:34:07Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:28.317723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-64c1bc42-615e-4bd7-b6c0-b6e62f628072, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::189/64', '172.24.5.221/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:28.317723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=64c1bc42-615e-4bd7-b6c0-b6e62f628072, lrouter_port=lrp-64c1bc42-615e-4bd7-b6c0-b6e62f628072, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:28.317723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.320594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:28.320594 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:28.320594 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:28.320594 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:28.320594 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:28.320594 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:28.320594 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.320594 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:28.321807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 64c1bc42-615e-4bd7-b6c0-b6e62f628072 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.321807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:28.321807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:28.321807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:28.321807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:28.321807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:28.321807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:38:28.321807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:28.321807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:28.321807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:28.321807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:28.322135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:28.322135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:28.322135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:28.322135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:28.322135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:28.322135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:28.322135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:28.322135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:28.322135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:28.322135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:28.322135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:28.322135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:28.322135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:28.322135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:28.323532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:28.323532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.323532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:28.323532 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 91ac5e2c-0c45-47b0-a854-6f34ff4755d1 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:28.442630 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-91ac5e2c-0c45-47b0-a854-6f34ff4755d1, resource={'id': '91ac5e2c-0c45-47b0-a854-6f34ff4755d1', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:fe:6a:63', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '7f17481f-41b4-4cca-9c5f-2f20e531ae95', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1189, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.34'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::23'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:37:01Z', 'updated_at': '2026-06-01T11:37:07Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:28.446186 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-91ac5e2c-0c45-47b0-a854-6f34ff4755d1, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::23/64', '172.24.5.34/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:28.450615 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=91ac5e2c-0c45-47b0-a854-6f34ff4755d1, lrouter_port=lrp-91ac5e2c-0c45-47b0-a854-6f34ff4755d1, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:28.450615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.450615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:28.450615 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:28.450615 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:28.450615 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:28.450615 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:28.450615 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:28.450615 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.450615 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:28.452219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 91ac5e2c-0c45-47b0-a854-6f34ff4755d1 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.452219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:28.452219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:28.452219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:28.452219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:28.452219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:28.452219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:38:28.452219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:28.452219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:28.452219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:28.452219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:28.452823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:28.452823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:28.452823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:28.452823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:28.452823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:28.452823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:28.452823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:28.452823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:28.452823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:28.452823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:28.452823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:28.452823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:28.452823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:28.452823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:28.453273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:28.453273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.453273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:28.453273 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource b76e8c43-e6a5-4e0e-ae38-a143937e6670 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:38:28.600072 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-b76e8c43-e6a5-4e0e-ae38-a143937e6670, resource={'id': 'b76e8c43-e6a5-4e0e-ae38-a143937e6670', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:3a:17:e7', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '065f5984-bcd4-48bd-b8cb-3119f0d33351', 'device_owner': 'network:router_gateway', 'standard_attr_id': 977, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.173'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::64'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:35:10Z', 'updated_at': '2026-06-01T11:35:13Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:28.600238 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-b76e8c43-e6a5-4e0e-ae38-a143937e6670, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.173/24', '2001:db8::64/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:28.600238 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=b76e8c43-e6a5-4e0e-ae38-a143937e6670, lrouter_port=lrp-b76e8c43-e6a5-4e0e-ae38-a143937e6670, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:28.600238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.600420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:28.600420 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:28.600420 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:28.600420 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:28.600420 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:28.600420 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:28.600420 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.600420 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:28.601105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource b76e8c43-e6a5-4e0e-ae38-a143937e6670 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.601105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:28.601105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:38:28.601105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:38:28.601105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:38:28.601105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:38:28.601105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:38:28.601105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:28.601105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:28.601105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:28.601105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:28.601362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:28.601362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:28.601362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:28.601362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:28.601362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:28.601362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:28.601362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:28.601362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:28.601362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:28.601362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:28.601362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:28.601362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:28.601362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:28.601362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:28.601672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:28.601672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:28.601672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:28.601672 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 27175b68-b6e6-4311-a3af-4ef5317e3a54 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:38:28.601672 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(27175b68-b6e6-4311-a3af-4ef5317e3a54) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:38:28.608184 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 50d164e5-522a-454c-a3b6-7c58f6d7f8c0 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:38:28.609131 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(50d164e5-522a-454c-a3b6-7c58f6d7f8c0) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:38:28.615929 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 551f6501-c620-4497-99e7-f711beeb1a83 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:38:28.615929 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(551f6501-c620-4497-99e7-f711beeb1a83) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:38:28.623758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 5cc62699-5246-4352-975c-404e38723865 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:38:28.624351 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(5cc62699-5246-4352-975c-404e38723865) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:38:28.637975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 69d9dd8d-c44f-40ce-bbd8-826dd81f4e7b (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:38:28.638243 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(69d9dd8d-c44f-40ce-bbd8-826dd81f4e7b) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:38:28.647908 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 8ad4fd9b-b654-4621-ab9b-dac381c5d3a2 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:38:28.648219 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(8ad4fd9b-b654-4621-ab9b-dac381c5d3a2) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:38:28.656720 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource b0ebedd5-d660-42dd-a1eb-38d516dc0a53 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:38:28.657113 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(b0ebedd5-d660-42dd-a1eb-38d516dc0a53) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:38:28.665973 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource b44895a8-10ac-44b4-ba7e-e3d8b565ab74 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:38:28.668229 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(b44895a8-10ac-44b4-ba7e-e3d8b565ab74) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:38:28.673703 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource e1377bdc-d382-4e96-8f17-c858f5f48b9e (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:38:28.673703 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(e1377bdc-d382-4e96-8f17-c858f5f48b9e) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:38:28.679996 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource e6f852c7-9dab-4a79-8d53-344636602b2e (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:38:28.680303 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(e6f852c7-9dab-4a79-8d53-344636602b2e) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:38:28.687994 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource ea6248bd-6d44-441b-a8fc-e1453ecced34 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:38:28.688237 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(ea6248bd-6d44-441b-a8fc-e1453ecced34) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:38:28.694133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource eb5f7b5c-78f7-43b7-970b-8352e4938241 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:38:28.694420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(eb5f7b5c-78f7-43b7-970b-8352e4938241) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:38:28.700829 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource ed4c2940-4773-465b-bffa-d8fb794131f1 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:38:28.701078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(ed4c2940-4773-465b-bffa-d8fb794131f1) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:38:28.708819 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource f05a2d8b-6e41-462f-8ddc-585388698387 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:38:28.709104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(f05a2d8b-6e41-462f-8ddc-585388698387) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:38:28.714967 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_for_inconsistencies finished in 1.639 seconds Jun 01 11:38:31.869364 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:31.872454 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:31.872454 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:31.872454 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:31.872454 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:31.872454 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:31.872817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:31.872817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:31.872817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:31.872817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:31.872817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:31.873458 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:31.873458 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:31.873458 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-741a0ca0-b62e-469e-b080-ea35c1e80ef1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:31.873458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.873458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:31.873458 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:31.873458 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:31.873458 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:31.873458 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:31.873458 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:31.873458 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.873458 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:31.874208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.874208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:31.874208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:31.874208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:31.874208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.874208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:31.874208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:31.874208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.874208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:31.874208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:31.874208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:31.874208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.874590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:31.874590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:31.874590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:31.874590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:31.874590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:31.874590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:31.874590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:31.874590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:31.874590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:31.874590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:31.874590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:31.874590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:31.874590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:31.874955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.874955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:31.875348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.875348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:31.875348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:31.875348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:31.875348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:31.875348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:31.875348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.875348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:31.875348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:31.875348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.875348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:31.875348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:31.875348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.875348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:31.875348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:31.875348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:31.875812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:31.876295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:31.876295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.876295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:31.905003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:31.905522 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:31.905645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:31.905762 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:38:31.906118 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:31.906318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.906846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:31.906846 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:31.906846 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:31.906846 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:31.906846 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:31.906846 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:31.906846 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.906846 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:31.907567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.907567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:31.907567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:31.907567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:31.907567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.907567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:31.907567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:31.907567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:31.907567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.907567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:31.907567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:31.907567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:31.907567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:31.907567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:31.907567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:31.907850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:31.907850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:31.907850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:31.907850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:31.907850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:31.907850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:31.907850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:31.907850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.907850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:31.908278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.908278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:31.908278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:31.908278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:31.908278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.908278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:31.908278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:31.908278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:38:31.908278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:31.908278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:31.908278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.908278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:31.908488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:31.908488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:31.908488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.908488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:31.908488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:31.908488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:31.908488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:31.908488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:31.908488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:31.908488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:31.908488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:31.908488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:31.908488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:31.908488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:31.908740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:31.908740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:31.908740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.908740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:31.909140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.909140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:31.909140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:31.909140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:31.909140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:31.909140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:31.909140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.909140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:31.909140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:31.909140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.909140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:31.909140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:31.909140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.909140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:31.909140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:31.909140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:31.909403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:31.909757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:31.909757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:31.909757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:31.909757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:31.909757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:31.909757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.909757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:31.940417 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:31.944959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:31.945276 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:31.945701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:31.947931 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:31.948165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.948747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:31.948747 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:31.948747 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:31.948747 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:31.948747 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:31.948747 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:31.948747 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.948747 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:31.949951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.949951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:31.949951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:31.949951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:31.949951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.949951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:31.949951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:31.949951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:31.949951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.949951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:31.949951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:31.949951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:31.949951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:31.949951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:31.949951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:31.950304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:31.950304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:31.950304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:31.950304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:31.950304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:31.950304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:31.950304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:31.950304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.950304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:31.951386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.951386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:31.951386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:31.951386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:31.951386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.951386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:31.951386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:31.951386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:31.951386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:31.951386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.951386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:31.951698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:31.951698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:31.951698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.951698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:31.951698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:31.951698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:31.951698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:31.951698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:31.951698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:31.951698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:31.951698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:31.951698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:31.951698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:31.951698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:31.952019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:31.952019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:31.952019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.952019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.953241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:31.953821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:31.954299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:31.954299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:31.954299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.954299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:31.977777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:31.978118 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:31.978256 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:31.978451 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:31.978985 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:31.979209 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:31.979505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.980099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:31.980099 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:31.980099 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:31.980099 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:31.980099 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:31.980099 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:31.980099 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.980099 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:31.981312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.981312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:31.981312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:31.981312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:31.981312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.981312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:31.981312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:31.981312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.981312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:31.981312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:31.981312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:31.981312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.981600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:31.981600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:31.981600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:31.981600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:31.981600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:31.981600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:31.981600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:31.981600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:31.981600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:31.981600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:31.981600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:31.981600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:31.981600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:31.981858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.981858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:31.983693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.983693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:31.983693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:31.983693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:31.983693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:31.983693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:31.983693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.983693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:31.983693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:31.983693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.983693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:31.983693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:31.983693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.983693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:31.983693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:31.983693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:31.984226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:31.984808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:31.984808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:31.984808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:32.006675 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:32.007033 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:32.007169 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:32.008089 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:32.058641 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:32.058935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:32.059979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:32.059979 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:32.059979 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:32.059979 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:32.059979 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:32.059979 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:32.059979 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:32.059979 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:32.060941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.053 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:32.060941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:32.060941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:32.060941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:32.060941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:32.060941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:32.060941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:32.060941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:32.060941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:32.060941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:32.060941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:32.060941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:32.061324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:32.061324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:32.061324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:32.061324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:32.061324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:32.061324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:32.061324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:32.061324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:32.061324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:32.061324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:32.061324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:32.061324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:32.061324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:32.061665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:32.061665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:32.062679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:32.062679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:32.062679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:32.062679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:32.062679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:32.062679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:32.062679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:32.062679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:32.062679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:32.062679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:32.062679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:32.062679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:32.062679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:32.062679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:32.062679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:32.062679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:32.063155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:32.063729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:32.063729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:32.063729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:36.870044 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:36.870489 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:36.870489 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:36.871901 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:36.872125 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:36.872331 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:36.872562 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:36.873500 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:36.873967 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:36.874883 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:36.875103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:36.875319 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:36.875699 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:36.875914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-741a0ca0-b62e-469e-b080-ea35c1e80ef1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:36.876802 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-044e825b-f1ca-4149-8ffc-6085a8ae6e1a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:36.877048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.878240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:36.878240 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:36.878240 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:36.878240 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:36.878240 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:36.878240 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:36.878240 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.878240 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:36.879281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.879281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:36.879281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:36.879281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:36.879281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.879281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:36.879281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:36.879281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.879281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:36.879281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:36.879281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:36.879281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.879755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:36.879755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:36.879755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:36.879755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:36.879755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:36.879755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:36.879755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:36.879755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:36.879755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:36.879755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:36.879755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:36.879755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:36.879755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:36.880060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.880060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:36.880305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.880305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:36.880305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:36.880305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:36.880305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:36.880305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:36.880305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.880305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:36.880305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:36.880305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.880305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:36.880305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:36.880305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.880305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:36.880305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:36.880305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:36.880765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:36.881229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:36.881229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.881229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:36.905798 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:36.906126 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:36.906291 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:36.906418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:38:36.907197 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:36.907501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.908152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:36.908152 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:36.908152 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:36.908152 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:36.908152 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:36.908152 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:36.908152 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.908152 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:36.909286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.909286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:36.909286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:36.909286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:36.909286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.909286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:36.909286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:36.909286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:36.909286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.909286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:36.909286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:36.909286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:36.909286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:36.909286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:36.909286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:36.909991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:36.909991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:36.909991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:36.909991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:36.909991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:36.909991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:36.909991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:36.909991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.909991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:36.910284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.910284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:36.910284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:36.910284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:36.910284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.910284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:36.910284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:36.910284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:38:36.910284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:36.910284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:36.910284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.910284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:36.910707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:36.910707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:36.910707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.910707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:36.910707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:36.910707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:36.910707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:36.910707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:36.910707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:36.910707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:36.910707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:36.910707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:36.910707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:36.910707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:36.911146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:36.911146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:36.911146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.911146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:36.911932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.911932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:36.911932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:36.911932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:36.911932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:36.911932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:36.911932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.911932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:36.911932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:36.911932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.911932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:36.911932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:36.911932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.911932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:36.911932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:36.911932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:36.912459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:36.913004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:36.913004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:36.913004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:36.913004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:36.913004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:36.913004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.913004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:36.944566 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:36.944859 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:36.944983 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:36.945121 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:36.946042 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:36.946377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.947117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:36.947117 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:36.947117 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:36.947117 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:36.947117 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:36.947117 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:36.947117 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.947117 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:36.948142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.948142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:36.948142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:36.948142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:36.948142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.948142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:36.948142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:36.948142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:36.948142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.948142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:36.948142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:36.948142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:36.948142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:36.948142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:36.948142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:36.948657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:36.948657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:36.948657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:36.948657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:36.948657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:36.948657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:36.948657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:36.948657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.948657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:36.949577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.949577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:36.949577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:36.949577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:36.949577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.949577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:36.949577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:36.949577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:36.949577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:36.949577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.949577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:36.949941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:36.949941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:36.949941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.949941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:36.949941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:36.949941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:36.949941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:36.949941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:36.949941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:36.949941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:36.949941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:36.949941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:36.949941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:36.949941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:36.950335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:36.950335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:36.950335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.950335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.951375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:36.951927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:36.952401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:36.952401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:36.952401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.952401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:36.981779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:36.982330 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:36.982449 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:36.982668 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:36.983137 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:36.983385 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:36.983801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.984696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:36.984696 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:36.984696 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:36.984696 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:36.984696 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:36.984696 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:36.984696 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.984696 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:36.985686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.985686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:36.985686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:36.985686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:36.985686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.985686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:36.985686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:36.985686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.985686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:36.985686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:36.985686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:36.985686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.986091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:36.986091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:36.986091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:36.986091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:36.986091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:36.986091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:36.986091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:36.986091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:36.986091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:36.986091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:36.986091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:36.986091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:36.986091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:36.986524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.986524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:36.986524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.986524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:36.986524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:36.986524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:36.986524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:36.986524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:36.986524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.986524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:36.986524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:36.986524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.986524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:36.986524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:36.986524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:36.987152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:36.987958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:36.987958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:36.987958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:36.987958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:36.987958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:36.987958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:37.008145 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:37.008145 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:37.008145 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:37.008298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:37.062365 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:37.062578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:37.063160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:37.063160 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:37.063160 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:37.063160 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:37.063160 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:37.063160 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:37.063160 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:37.063160 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:37.064030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:37.064030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:37.064030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:37.064030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:37.064030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:37.064030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:37.064030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:37.064030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:37.064030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:37.064030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:37.064030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:37.064030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:37.064267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:37.064267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:37.064267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:37.064267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:37.064267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:37.064267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:37.064267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:37.064267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:37.064267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:37.064267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:37.064267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:37.064267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:37.064267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:37.064500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:37.064500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:37.065183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:37.065183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:37.065183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:37.065183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:37.065183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:37.065183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:37.065183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:37.065183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:37.065183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:37.065183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:37.065183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:37.065183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:37.065183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:37.065183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:37.065183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:37.065183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:37.065459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:37.065872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:37.065872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:37.065872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:41.870421 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:41.872001 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:41.872001 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:41.872418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:41.872683 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:41.872825 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:41.872967 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:41.873376 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:41.873376 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:41.873376 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:41.873376 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:41.873523 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:41.873648 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:41.873746 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-741a0ca0-b62e-469e-b080-ea35c1e80ef1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:41.873845 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-044e825b-f1ca-4149-8ffc-6085a8ae6e1a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:41.873981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.876084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:41.876084 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:41.876084 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:41.876084 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:41.876084 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:41.876084 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:41.876084 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.876084 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:41.876084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.876084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:41.876084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:41.876084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:41.876084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.876084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:41.876084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:41.876437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.876437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:41.876437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:41.876437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:41.876437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.876437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:41.876437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:41.876437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:41.876437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:41.876437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:41.876437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:41.876437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:41.876437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:41.876437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:41.876772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:41.876772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:41.876772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:41.876772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:41.876772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.876772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:41.876772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.876772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:41.876772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:41.876772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:41.876772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:41.876772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:41.876772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.876772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:41.877517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:41.877965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:41.877965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:41.877965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:41.877965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:41.877965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:41.877965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:41.877965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:41.877965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:41.877965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.877965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:41.906169 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:41.906640 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:41.906835 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:41.907006 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:38:41.907712 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:41.907985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.908588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:41.908588 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:41.908588 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:41.908588 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:41.908588 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:41.908588 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:41.908588 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.908588 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:41.910743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.910743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:41.910743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:41.910743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:41.910743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.910743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:41.910743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:41.910743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:41.910743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.910743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:41.910743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:41.910743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:41.910743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:41.910743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:41.910743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:41.911081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:41.911081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:41.911081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:41.911081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:41.911081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:41.911081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:41.911081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:41.911081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.911081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:41.911081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.911081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:41.911081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:41.911081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:41.911081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.911339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:41.911339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:41.911339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:38:41.911339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:41.911339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:41.911339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.911339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:41.911339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:41.911339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:41.911339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.911339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:41.911339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:41.911339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:41.911590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:41.911590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:41.911590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:41.911590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:41.911590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:41.911590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:41.911590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:41.911590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:41.911590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:41.911590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:41.911590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.911590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:41.911797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.911797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:41.911797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:41.911797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:41.911797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:41.911797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:41.911797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.911797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:41.911797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:41.911797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.911797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:41.911797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:41.911797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.911797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:41.911797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:41.911797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:41.912077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:41.912375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:41.912375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:41.912375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:41.912375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:41.912375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:41.912375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.912375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:41.944967 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:41.945380 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:41.945638 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:41.945833 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:41.946403 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:41.946710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.949216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:41.949624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:41.949624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:41.949624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:41.949624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:41.949624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:41.949624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:41.949624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:41.949624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:41.949624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:41.949624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:41.949624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:41.949624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:41.949624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.949624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:41.949624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.949887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:41.949887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:41.949887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:41.949887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.949887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:41.949887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:41.949887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:41.949887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:41.949887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.949887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:41.949887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:41.949887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:41.949887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.950118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:41.950118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:41.950118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:41.950118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:41.950118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:41.950118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:41.950118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:41.950118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:41.950118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:41.950118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:41.950118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:41.950118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:41.950118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:41.950369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.950369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:41.950369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.950369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:41.950369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:41.950369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:41.950369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:41.950369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:41.950369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.950369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:41.950369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:41.950369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.950369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:41.950369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:41.950369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:41.950709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:41.951033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:41.951033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:41.951033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:41.951033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:41.951033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:41.951033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:41.951033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.951033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:41.982435 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:41.982881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:41.983093 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:41.983307 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:41.983880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:41.984136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:41.984423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.986392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:41.986392 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:41.986392 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:41.986392 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:41.986392 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:41.986392 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:41.986392 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.986392 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:41.986392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.986392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:41.986392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:41.986392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:41.986392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.986392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:41.986392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:41.986731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.986731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:41.986731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:41.986731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:41.986731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.986731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:41.986731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:41.986731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:41.986731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:41.986731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:41.986731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:41.986731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:41.986731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:41.986731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:41.986946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:41.986946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:41.986946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:41.986946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:41.986946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.986946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:41.986946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.986946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:41.986946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:41.986946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:41.986946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:41.986946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:41.986946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.986946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:41.987196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:41.987524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:41.987524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:41.987524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:41.987524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:41.987524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:41.987524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:41.987524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:41.987524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:41.987524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:41.987524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:42.007603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:42.008020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:42.008223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:42.008939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:42.050292 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:42.050517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:42.053354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:42.053354 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:42.053354 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:42.053354 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:42.053354 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:42.053354 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:42.053354 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:42.053354 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:42.053354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:42.053354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:42.053354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:42.053354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:42.053354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:42.053354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:42.053354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:42.053854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:42.053854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:42.053854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:42.053854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:42.053854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:42.053854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:42.053854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:42.053854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:42.053854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:42.053854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:42.053854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:42.053854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:42.053854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:42.053854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:42.054234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:42.054234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:42.054234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:42.054234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:42.054234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:42.054234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:42.054234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:42.054234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:42.054234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:42.054234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:42.054234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:42.054234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:42.054234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:42.054234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:42.054639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:42.055175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:42.055175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:42.055175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:42.055175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:42.055175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:42.055175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:42.055175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:42.055175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:42.055175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:46.871298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:46.872330 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:46.872617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:46.885779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:46.885779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:46.885779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:46.885779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:46.885779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:46.886582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:46.886582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:46.886582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:46.886582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:46.886582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-741a0ca0-b62e-469e-b080-ea35c1e80ef1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:46.887823 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-044e825b-f1ca-4149-8ffc-6085a8ae6e1a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:46.887823 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:46.887823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.887823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:46.887823 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:46.887823 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:46.887823 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:46.887823 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:46.887823 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:46.887823 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.887823 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:46.890964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.014 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.890964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:46.890964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:46.890964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:46.890964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.890964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:46.890964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:46.890964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.890964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:46.890964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:46.890964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:46.890964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.891339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:46.891339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:46.891339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:46.891339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:46.891339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:46.891339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:46.891339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:46.891339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:46.891339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:46.891339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:46.891339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:46.891339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:46.891339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:46.891740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.891740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:46.891740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.891740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:46.891740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:46.891740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:46.891740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:46.891740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:46.891740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.891740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:46.891740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:46.891740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.891740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:46.891740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:46.891740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:46.892148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:46.892680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:46.892680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:46.892680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:46.892680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:46.892680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.892680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:46.906749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:46.907772 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:46.907772 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:46.907772 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:38:46.908167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:46.908354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.911207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:46.911659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:46.911659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:46.911659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:46.911659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:46.911659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:46.911659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:46.911659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:46.911659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:46.911659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:46.911659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:46.911659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:46.911659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:46.911659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.911659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:46.911659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.912001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:46.912001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:46.912001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:46.912001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.912001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:46.912001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:46.912001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:38:46.912001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:46.912001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:46.912001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.912001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:46.912001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:46.912001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:46.912289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.912289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:46.912289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:46.912289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:46.912289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:46.912289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:46.912289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:46.912289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:46.912289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:46.912289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:46.912289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:46.912289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:46.912289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:46.912289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:46.912793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.912793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:46.912793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.912793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:46.912793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:46.912793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:46.912793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:46.912793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:46.912793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.912793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:46.912793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:46.912793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.912793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:46.912793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:46.912793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:46.913240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:46.913845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:46.913845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:46.913845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:46.913845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:46.913845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:46.913845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:46.913845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:46.913845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.913845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:46.945734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:46.946654 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:46.946654 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:46.946654 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:46.946915 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:46.947106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.950603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:46.951205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:46.951205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:46.951205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:46.951205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:46.951205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:46.951205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:46.951205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:46.951205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:46.951205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:46.951205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:46.951205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:46.951205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:46.951205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.951205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:46.951205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.951609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:46.951609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:46.951609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:46.951609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.951609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:46.951609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:46.951609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:46.951609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:46.951609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.951609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:46.951609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:46.951609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:46.951609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.951936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:46.951936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:46.951936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:46.951936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:46.951936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:46.951936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:46.951936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:46.951936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:46.951936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:46.951936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:46.951936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:46.951936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:46.951936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:46.952354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.952354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.952978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:46.953428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:46.953939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:46.953939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:46.953939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.953939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:46.983084 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:46.983487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:46.983703 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:46.983924 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:46.984323 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:46.984525 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:46.984806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:46.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:46.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:46.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:46.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:46.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:46.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:46.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:46.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:46.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:46.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:46.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:46.986972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.986972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:46.986972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:46.986972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:46.986972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.986972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:46.986972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:46.986972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:46.986972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:46.986972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:46.986972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:46.986972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:46.986972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:46.986972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:46.987203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:46.987203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:46.987203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:46.987203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:46.987203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.987203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:46.987203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.987203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:46.987203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:46.987203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:46.987203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:46.987203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:46.987203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.987203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:46.987447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:46.987798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:46.987798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:46.987798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:46.987798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:46.987798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:46.987798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:46.987798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:46.987798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:46.987798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:46.987798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:47.008140 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:47.008553 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:47.008801 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:47.009506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:47.043591 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:47.043717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:47.045420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:47.045420 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:47.045420 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:47.045420 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:47.045420 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:47.045420 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:47.045420 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:47.045420 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:47.045420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:47.045420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:47.045420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:47.045420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:47.045420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:47.045420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:47.045420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:47.045814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:47.045814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:47.045814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:47.045814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:47.045814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:47.045814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:47.045814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:47.045814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:47.045814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:47.045814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:47.045814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:47.045814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:47.045814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:47.045814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:47.046095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:47.046095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:47.046095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:47.046095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:47.046095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:47.046095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:47.046095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:47.046095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:47.046095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:47.046095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:47.046095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:47.046095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:47.046095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:47.046095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:47.046323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:47.046777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:47.046777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:47.046777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:47.046777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:47.046777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:47.046777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:47.046777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:47.046777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:47.046777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:51.872433 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:51.873405 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:51.873580 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:51.875087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:51.875274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:51.875433 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:51.875633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:51.875877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:51.876053 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:51.876209 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:51.876364 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:51.876556 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:51.876722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-741a0ca0-b62e-469e-b080-ea35c1e80ef1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:51.876878 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-044e825b-f1ca-4149-8ffc-6085a8ae6e1a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:51.877027 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:51.877176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.878807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:51.878807 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:51.878807 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:51.878807 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:51.878807 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:51.878807 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:51.878807 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.878807 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:51.879776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.879776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:51.879776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:51.879776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:51.879776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.879776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:51.879776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:51.879776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.879776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:51.879776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:51.879776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:51.879776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.880324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:51.880324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:51.880324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:51.880324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:51.880324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:51.880324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:51.880324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:51.880324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:51.880324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:51.880324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:51.880324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:51.880324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:51.880324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:51.880776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.880776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:51.883128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.883128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:51.883128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:51.883128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:51.883128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:51.883128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:51.883128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.883128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:51.883128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:51.883128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.883128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:51.883128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:51.883128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.883128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:51.883128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:51.883128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:51.883715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:51.884381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:51.884381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.884381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:51.907688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:51.908453 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:51.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:51.908810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:38:51.909920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:51.910179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.910874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:51.910874 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:51.910874 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:51.910874 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:51.910874 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:51.910874 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:51.910874 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.910874 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:51.912800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.912800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:51.912800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:51.912800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:51.912800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.912800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:51.912800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:51.912800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:51.912800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.912800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:51.912800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:51.912800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:51.912800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:51.912800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:51.912800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:51.913291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:51.913291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:51.913291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:51.913291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:51.913291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:51.913291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:51.913291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:51.913291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.913291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:51.913861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.913861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:51.913861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:51.913861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:51.913861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.913861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:51.913861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:51.913861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:38:51.913861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:51.913861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:51.913861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.913861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:51.914227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:51.914227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:51.914227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.914227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:51.914227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:51.914227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:51.914227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:51.914227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:51.914227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:51.914227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:51.914227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:51.914227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:51.914227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:51.914227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:51.914662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:51.914662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:51.914662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.914662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:51.915025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.915025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:51.915025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:51.915025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:51.915025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:51.915025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:51.915025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.915025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:51.915025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:51.915025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.915025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:51.915025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:51.915025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.915025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:51.915025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:51.915025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:51.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:51.916006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:51.916006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:51.916006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:51.916006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:51.916006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:51.916006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.916006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:51.946292 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:51.946729 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:51.946870 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:51.947035 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:51.947453 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:51.947702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.948171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:51.948171 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:51.948171 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:51.948171 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:51.948171 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:51.948171 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:51.948171 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.948171 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:51.949234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.949234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:51.949234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:51.949234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:51.949234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.949234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:51.949234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:51.949234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:51.949234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.949234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:51.949234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:51.949234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:51.949234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:51.949234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:51.949234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:51.949530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:51.949530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:51.949530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:51.949530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:51.949530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:51.949530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:51.949530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:51.949530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.949530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:51.950240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.950240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:51.950240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:51.950240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:51.950240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.950240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:51.950240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:51.950240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:51.950240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:51.950240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.950240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:51.950526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:51.950526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:51.950526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.950526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:51.950526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:51.950526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:51.950526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:51.950526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:51.950526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:51.950526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:51.950526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:51.950526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:51.950526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:51.950526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:51.950782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:51.950782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:51.950782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.950782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.951445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:51.951816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:51.952171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:51.952171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:51.952171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.952171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:51.985857 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:51.986301 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:51.986462 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:51.986736 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:51.987198 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:51.987491 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:51.987879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.989815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:51.989815 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:51.989815 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:51.989815 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:51.989815 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:51.989815 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:51.989815 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.989815 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:51.990645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.990645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:51.990645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:51.990645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:51.990645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.990645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:51.990645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:51.990645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.990645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:51.990645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:51.990645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:51.990645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:51.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:51.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:51.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:51.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:51.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:51.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:51.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:51.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:51.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:51.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:51.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:51.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:51.993293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.993293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:51.993293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.993293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:51.993293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:51.993293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:51.993293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:51.993293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:51.993293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.993293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:51.993293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:51.993293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.993293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:51.993293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:51.993293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:51.993729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:51.994256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:51.994256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:51.994256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:51.994256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:51.994256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:51.994256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:52.008772 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:52.009187 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:52.009339 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:52.010165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:52.051068 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:52.051267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:52.051833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:52.051833 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:52.051833 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:52.051833 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:52.051833 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:52.051833 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:52.051833 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:52.051833 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:52.052548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:52.052548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:52.052548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:52.052548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:52.052548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:52.052548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:52.052548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:52.052548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:52.052548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:52.052548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:52.052548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:52.052548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:52.052832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:52.052832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:52.052832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:52.052832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:52.052832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:52.052832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:52.052832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:52.052832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:52.052832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:52.052832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:52.052832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:52.052832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:52.052832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:52.053102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:52.053102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:52.053331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:52.053331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:52.053331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:52.053331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:52.053331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:52.053331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:52.053331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:52.053331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:52.053331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:52.053331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:52.053331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:52.053331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:52.053331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:52.053331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:52.053331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:52.053331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:52.053661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:52.054005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:52.054005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:52.054005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:56.873695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:56.874236 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:56.874236 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:56.875377 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:56.875554 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:56.875756 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:56.875905 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:56.876058 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:56.876233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:56.876336 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:56.876470 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:56.876643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:56.876785 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-741a0ca0-b62e-469e-b080-ea35c1e80ef1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:56.876932 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-044e825b-f1ca-4149-8ffc-6085a8ae6e1a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:56.877075 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:56.877275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:56.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:56.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:56.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:56.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:56.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:56.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.878328 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:56.879521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.879521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:56.879521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:56.879521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:56.879521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.879521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:56.879521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:56.879521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.879521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:56.879521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:56.879521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:56.879521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.879986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:56.879986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:56.879986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:56.879986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:56.879986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:56.879986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:56.879986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:56.879986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:56.879986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:56.879986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:56.879986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:56.879986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:56.879986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:56.880322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.880322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:56.880757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.880757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:56.880757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:56.880757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:56.880757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:56.880757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:56.880757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.880757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:56.880757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:56.880757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.880757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:56.880757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:56.880757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.880757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:56.880757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:56.880757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:56.881103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:56.881384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:56.881384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.881384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:56.909105 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:56.909517 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:56.909676 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:56.909859 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:38:56.910345 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:56.910583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.911145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:56.911145 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:56.911145 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:56.911145 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:56.911145 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:56.911145 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:56.911145 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.911145 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:56.912101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.912101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:56.912101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:56.912101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:56.912101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.912101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:56.912101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:56.912101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:56.912101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.912101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:56.912101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:56.912101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:56.912101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:56.912101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:56.912101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:56.912509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:56.912509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:56.912509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:56.912509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:56.912509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:56.912509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:56.912509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:56.912509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.912509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:56.913053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.913053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:56.913053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:56.913053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:56.913053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.913053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:56.913053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:56.913053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:38:56.913053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:56.913053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:56.913053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.913053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:56.913391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:56.913391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:56.913391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.913391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:56.913391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:56.913391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:56.913391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:56.913391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:56.913391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:56.913391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:56.913391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:56.913391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:56.913391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:56.913391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:56.913718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:56.913718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:56.913718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.913718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:56.913718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.913718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:56.913718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:56.913718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:56.913718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:56.913718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:56.913718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.913718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:56.913718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:56.913718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:56.914059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:56.914488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:56.914488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:56.914488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:56.914488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:56.914488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:56.914488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:56.914488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:56.914488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:56.914488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:56.914488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:56.914488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.914488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:56.946746 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:56.947160 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:56.947268 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:56.947384 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:56.947824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:56.948051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.948609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:56.948609 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:56.948609 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:56.948609 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:56.948609 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:56.948609 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:56.948609 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.948609 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:56.949373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.949373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:38:56.949373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:56.949373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:56.949373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.949373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:56.949373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:56.949373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:38:56.949373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.949373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:38:56.949373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:56.949373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:38:56.949373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:56.949373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:38:56.949373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:38:56.949703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:56.949703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:38:56.949703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:56.949703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:38:56.949703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:38:56.949703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:56.949703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:38:56.949703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.949703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:38:56.950057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.950057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:56.950057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:56.950057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:56.950057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.950057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:56.950057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:56.950057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:56.950057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:56.950057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.950057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:56.950533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:56.950533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:56.950533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.950533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:56.950533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:56.950533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:56.950533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:56.950533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:56.950533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:56.950533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:56.950533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:56.950533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:56.950533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:56.950533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:56.951225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:56.951225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:56.951225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.951225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:56.951225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.951225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:56.951225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:56.951225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:56.951225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:56.951225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:56.951225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.951225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:56.951225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:56.951225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:56.951583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:56.951910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:56.951910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:56.951910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:56.951910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:56.951910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:56.951910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:56.951910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:56.951910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:56.951910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:56.951910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.951910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:56.986298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:56.986618 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:56.986746 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:56.986922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:56.987415 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:56.987670 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:56.987978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.988594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:56.988594 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:56.988594 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:56.988594 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:56.988594 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:56.988594 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:56.988594 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.988594 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:56.989663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.989663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:56.989663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:56.989663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:56.989663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.989663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:56.989663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:56.989663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.989663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:56.989663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:56.989663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:56.989663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.990120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:56.990120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:56.990120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:56.990120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:56.990120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:56.990120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:56.990120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:56.990120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:56.990120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:56.990120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:56.990120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:56.990120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:56.990120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:56.990464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.990464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:56.990957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.990957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:56.990957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:56.990957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:56.990957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:56.990957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:56.990957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.990957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:56.990957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:56.990957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.990957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:56.990957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:56.990957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.990957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:56.990957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:56.990957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:56.991695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:56.992280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:56.992280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:56.992280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:38:57.009665 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:38:57.009974 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:38:57.010150 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:38:57.010879 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:38:57.053023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:38:57.053214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:57.053713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:38:57.053713 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:57.053713 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:38:57.053713 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:38:57.053713 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:57.053713 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:38:57.053713 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:57.053713 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:38:57.054345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:57.054345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:38:57.054345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:57.054345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:38:57.054345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:57.054345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:57.054345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:57.054345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:57.054345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:57.054345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:57.054345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:38:57.054345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:57.054751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:38:57.054751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:57.054751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:38:57.054751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:57.054751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:38:57.054751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:38:57.054751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:57.054751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:38:57.054751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:57.054751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:38:57.054751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:38:57.054751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:57.054751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:38:57.055088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:57.055088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:38:57.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:57.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:38:57.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:38:57.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:38:57.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:38:57.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:38:57.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:57.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:38:57.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:38:57.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:57.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:38:57.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:38:57.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:38:57.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:38:57.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:38:57.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:38:57.056429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:38:57.057218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:38:57.057218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:38:57.057218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:01.874360 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:01.875062 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:01.875146 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:01.876796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:01.877015 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:01.877140 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:01.877382 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:01.877448 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:01.877532 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:01.877687 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:01.877806 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:01.877922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:01.878039 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-741a0ca0-b62e-469e-b080-ea35c1e80ef1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:01.878174 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-044e825b-f1ca-4149-8ffc-6085a8ae6e1a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:01.878294 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:01.878413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.879139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:01.879139 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:01.879139 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:01.879139 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:01.879139 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:01.879139 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:01.879139 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.879139 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:01.880320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.880320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:01.880320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:01.880320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:01.880320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.880320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:01.880320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:01.880320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.880320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:01.880320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:01.880320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:01.880320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.880766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:01.880766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:01.880766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:01.880766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:01.880766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:01.880766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:01.880766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:01.880766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:01.880766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:01.880766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:01.880766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:01.880766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:01.880766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:01.881089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.881089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:01.881633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.881633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:01.881633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:01.881633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:01.881633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:01.881633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:01.881633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.881633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:01.881633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:01.881633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.881633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:01.881633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:01.881633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.881633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:01.881633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:01.881633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:01.882024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:01.882630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:01.882630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.882630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:01.909605 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:01.909910 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:01.909984 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:01.910126 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:39:01.910523 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:01.910750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.911201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:01.911201 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:01.911201 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:01.911201 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:01.911201 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:01.911201 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:01.911201 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.911201 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:01.911866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.911866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:01.911866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:01.911866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:01.911866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.911866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:01.911866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:01.911866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:01.911866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.911866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:01.911866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:01.911866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:01.911866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:01.911866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:01.911866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:01.912299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:01.912299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:01.912299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:01.912299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:01.912299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:01.912299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:01.912299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:01.912299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.912299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:01.912756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.912756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:01.912756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:01.912756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:01.912756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.912756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:01.912756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:01.912756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:39:01.912756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:01.912756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:01.912756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.912756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:01.913085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:01.913085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:01.913085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.913085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:01.913085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:01.913085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:01.913085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:01.913085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:01.913085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:01.913085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:01.913085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:01.913085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:01.913085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:01.913085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:01.913464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:01.913464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:01.913464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.913464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:01.913700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.913700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:01.913700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:01.913700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:01.913700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:01.913700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:01.913700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.913700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:01.913700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:01.913700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.913700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:01.913700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:01.913700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.913700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:01.913700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:01.913700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:01.914150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:01.914684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:01.914684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:01.914684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:01.914684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:01.914684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:01.914684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.914684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:01.947232 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:01.947577 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:01.947716 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:01.947896 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:01.948500 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:01.948775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.949279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:01.949279 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:01.949279 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:01.949279 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:01.949279 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:01.949279 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:01.949279 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.949279 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:01.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:01.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:01.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:01.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:01.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:01.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:01.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:01.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:01.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:01.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:01.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:01.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:01.950400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:01.950400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:01.950400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:01.950400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:01.950400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:01.950400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:01.950400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:01.950400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.950400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:01.950962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.950962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:01.950962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:01.950962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:01.950962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.950962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:01.950962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:01.950962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:01.950962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:01.950962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.950962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:01.951404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:01.951404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:01.951404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.951404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:01.951404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:01.951404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:01.951404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:01.951404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:01.951404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:01.951404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:01.951404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:01.951404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:01.951404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:01.951404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:01.951871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:01.951871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:01.951871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.951871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:01.951871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.951871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:01.951871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:01.951871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:01.951871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:01.951871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:01.951871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.951871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:01.951871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:01.951871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:01.952259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:01.952959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:01.952959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:01.952959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:01.952959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:01.952959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:01.952959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:01.952959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:01.952959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:01.952959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:01.952959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.952959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:01.986785 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:01.987119 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:01.987244 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:01.987406 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:01.987808 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:01.988094 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:01.988525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.989109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:01.989109 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:01.989109 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:01.989109 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:01.989109 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:01.989109 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:01.989109 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.989109 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:01.989810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.989810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:01.989810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:01.989810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:01.989810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.989810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:01.989810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:01.989810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.989810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:01.989810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:01.989810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:01.989810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.990244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:01.990244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:01.990244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:01.990244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:01.990244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:01.990244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:01.990244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:01.990244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:01.990244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:01.990244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:01.990244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:01.990244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:01.990244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:01.990589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.990589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:01.990715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.990715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:01.990715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:01.990715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:01.990715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:01.990715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:01.990715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.990715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:01.990715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:01.990715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.990715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:01.990715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:01.990715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.990715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:01.990715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:01.990715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:01.991132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:01.991520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:01.991520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:01.991520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:02.010190 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:02.010631 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:02.010781 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:02.011518 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:02.060271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:02.060444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:02.061267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:02.061267 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:02.061267 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:02.061267 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:02.061267 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:02.061267 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:02.061267 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:02.061267 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:02.062068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.051 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:02.062068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:02.062068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:02.062068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:02.062068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:02.062068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:02.062068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:02.062068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:02.062068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:02.062068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:02.062068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:02.062068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:02.062310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:02.062310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:02.062310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:02.062310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:02.062310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:02.062310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:02.062310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:02.062310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:02.062310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:02.062310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:02.062310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:02.062310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:02.062310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:02.062590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:02.062590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:02.063075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:02.063075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:02.063075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:02.063075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:02.063075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:02.063075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:02.063075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:02.063075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:02.063075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:02.063075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:02.063075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:02.063075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:02.063075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:02.063075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:02.063075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:02.063075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:02.063363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:02.063704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:02.063704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:02.063704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:06.875045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:06.876194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:06.877433 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:06.877827 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:06.878092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:06.878312 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:06.878452 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:06.878904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:06.878904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:06.878904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:06.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:06.879077 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-741a0ca0-b62e-469e-b080-ea35c1e80ef1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:06.879224 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-044e825b-f1ca-4149-8ffc-6085a8ae6e1a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:06.879285 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:06.879365 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:06.879495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.880711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:06.880711 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:06.880711 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:06.880711 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:06.880711 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:06.880711 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:06.880711 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.880711 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:06.884085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.884085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:06.884085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:06.884085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:06.884085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.884085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:06.884085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:06.884085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.884085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:06.884085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:06.884085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:06.884085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:06.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:06.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:06.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:06.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:06.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:06.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:06.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:06.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:06.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:06.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:06.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:06.884716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:06.885234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.885234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:06.887132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.887132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:06.887132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:06.887132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:06.887132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:06.887132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:06.887132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.887132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:06.887132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:06.887132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.887132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:06.887132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:06.887132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.887132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:06.887132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:06.887132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:06.888135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:06.889161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:06.889161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.889161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:06.913283 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:06.913863 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:06.914397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:06.914397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:39:06.914835 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:06.915039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.918088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:06.918088 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:06.918088 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:06.918088 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:06.918088 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:06.918088 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:06.918088 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.918088 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:06.922242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.922242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:06.922242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:06.922242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:06.922242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.922242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:06.922242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:06.922242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:06.922242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.922242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:06.922242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:06.922242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:06.922242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:06.922242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:06.922242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:06.923862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:06.923862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:06.923862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:06.923862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:06.923862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:06.923862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:06.923862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:06.923862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.923862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:06.923862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.923862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:06.923862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:06.923862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:06.923862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.924733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:06.924733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:06.924733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:39:06.924733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:06.924733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:06.924733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.924733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:06.924733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:06.924733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:06.924733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.924733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:06.924733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:06.924733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:06.925082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:06.925082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:06.925082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:06.925082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:06.925082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:06.925082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:06.925082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:06.925082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:06.925082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:06.925082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:06.925082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.925082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:06.925359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.925359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:06.925359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:06.925359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:06.925359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:06.925359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:06.925359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.925359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:06.925359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:06.925359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.925359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:06.925359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:06.925359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.925359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:06.925359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:06.925359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:06.925768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:06.926336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:06.926336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:06.926336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:06.926336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:06.926336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:06.926336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.926336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:06.948416 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:06.948944 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:06.949072 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:06.949218 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:06.949749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:06.950027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.952585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:06.953197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:06.953197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:06.953197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:06.953197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:06.953197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:06.953197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:06.953197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:06.953197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:06.953197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:06.953197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:06.953197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:06.953197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:06.953197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.953197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:06.953197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.953786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:06.953786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:06.953786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:06.953786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.953786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:06.953786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:06.953786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:06.953786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:06.953786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.953786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:06.953786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:06.953786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:06.953786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.954150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:06.954150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:06.954150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:06.954150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:06.954150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:06.954150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:06.954150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:06.954150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:06.954150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:06.954150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:06.954150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:06.954150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:06.954150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:06.954493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.954493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:06.954493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.954493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:06.954493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:06.954493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:06.954493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:06.954493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:06.954493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.954493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:06.954493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:06.954493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.954493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:06.954493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:06.954493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:06.954878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:06.955378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:06.955378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:06.955378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:06.955378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:06.955378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:06.955378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:06.955378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.955378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:06.987322 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:06.988397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:06.988397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:06.988397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:06.988845 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:06.989179 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:06.989556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.991983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:06.991983 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:06.991983 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:06.991983 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:06.991983 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:06.991983 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:06.991983 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.991983 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:06.991983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.991983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:06.991983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:06.991983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:06.991983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.991983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:06.991983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:06.992555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.992555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:06.992555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:06.992555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:06.992555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.992555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:06.992555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:06.992555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:06.992555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:06.992555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:06.992555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:06.992555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:06.992555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:06.992555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:06.992821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:06.992821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:06.992821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:06.992821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:06.992821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.992821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:06.992821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.992821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:06.992821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:06.992821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:06.992821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:06.992821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:06.992821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.992821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:06.993142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:06.993795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:06.993795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:06.993795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:06.993795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:06.993795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:06.993795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:06.993795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:06.993795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:06.993795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:06.993795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:07.010677 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:07.012193 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:07.012432 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:07.013259 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:07.076777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:07.077381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:07.078232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:07.078232 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:07.078232 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:07.078232 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:07.078232 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:07.078232 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:07.078232 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:07.078232 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:07.078758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.065 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:07.078758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:07.078758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:07.078758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:07.078758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:07.078758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:07.078758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:07.078758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:07.078758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:07.078758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:07.078758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:07.078758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:07.079187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:07.079187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:07.079187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:07.079187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:07.079187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:07.079187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:07.079187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:07.079187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:07.079187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:07.079187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:07.079187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:07.079187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:07.079187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:07.079430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:07.079430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:07.079562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:07.079562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:07.079562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:07.079562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:07.079562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:07.079562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:07.079562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:07.079562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:07.079562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:07.079562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:07.079562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:07.079562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:07.079562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:07.079562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:07.079562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:07.079562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:07.079857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:07.080167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:07.080167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:07.080167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:11.876324 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:11.876802 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:11.877179 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:11.878796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:11.879029 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:11.879197 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:11.879352 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:11.879565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:11.879714 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:11.879864 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:11.880117 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:11.880180 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-741a0ca0-b62e-469e-b080-ea35c1e80ef1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:11.880295 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-044e825b-f1ca-4149-8ffc-6085a8ae6e1a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:11.880435 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:11.880601 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:11.880745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.881296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:11.881296 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:11.881296 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:11.881296 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:11.881296 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:11.881296 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:11.881296 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.881296 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:11.882205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.882205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:11.882205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:11.882205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:11.882205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.882205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:11.882205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:11.882205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.882205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:11.882205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:11.882205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:11.882205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.882453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:11.882453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:11.882453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:11.882453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:11.882453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:11.882453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:11.882453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:11.882453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:11.882453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:11.882453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:11.882453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:11.882453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:11.882453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:11.882709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.882709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:11.882830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.882830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:11.882830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:11.882830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:11.882830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:11.882830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:11.882830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.882830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:11.882830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:11.882830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.882830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:11.882830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:11.882830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.882830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:11.882830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:11.882830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:11.883114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:11.883433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:11.883433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.883433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:11.935632 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:11.936645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:11.936645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:11.936645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:39:11.941506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:11.941790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.942356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:11.942356 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:11.942356 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:11.942356 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:11.942356 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:11.942356 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:11.942356 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.942356 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:11.943069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.943069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:11.943069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:11.943069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:11.943069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.943069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:11.943069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:11.943069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:11.943069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.943069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:11.943069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:11.943069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:11.943069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:11.943069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:11.943069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:11.943355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:11.943355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:11.943355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:11.943355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:11.943355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:11.943355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:11.943355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:11.943355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.943355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:11.943879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.943879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:11.943879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:11.943879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:11.943879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.943879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:11.943879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:11.943879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:39:11.943879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:11.943879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:11.943879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.943879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:11.944136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:11.944136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:11.944136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.944136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:11.944136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:11.944136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:11.944136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:11.944136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:11.944136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:11.944136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:11.944136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:11.944136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:11.944136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:11.944136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:11.944344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:11.944344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:11.944344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.944344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:11.944710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.944710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:11.944710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:11.944710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:11.944710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:11.944710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:11.944710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.944710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:11.944710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:11.944710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.944710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:11.944710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:11.944710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.944710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:11.944710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:11.944710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:11.944991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:11.945290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:11.945290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:11.945290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:11.945290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:11.945290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:11.945290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.945290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:11.949625 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:11.949981 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:11.950067 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:11.950170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:11.956768 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:11.956768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.956768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:11.956768 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:11.956768 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:11.956768 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:11.956768 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:11.956768 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:11.956768 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.956768 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:11.958553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.958553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:11.958553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:11.958553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:11.958553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.958553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:11.958553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:11.958553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:11.958553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.958553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:11.958553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:11.958553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:11.958553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:11.958553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:11.958553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:11.958926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:11.958926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:11.958926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:11.958926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:11.958926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:11.958926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:11.958926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:11.958926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.958926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:11.958926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.958926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:11.958926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:11.958926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:11.958926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.959260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:11.959260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:11.959260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:11.959260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:11.959260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.959260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:11.959260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:11.959260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:11.959260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.959260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:11.959260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:11.959260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:11.959260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:11.959520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:11.959520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:11.959520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:11.959520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:11.959520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:11.959520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:11.959520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:11.959520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:11.959520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:11.959520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.959520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.964342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:11.964823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:11.965302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:11.965302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:11.965302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.965302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:11.988066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:11.988445 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:11.988648 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:11.989640 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:11.990388 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:11.990736 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:11.991177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.992527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:11.992527 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:11.992527 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:11.992527 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:11.992527 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:11.992527 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:11.992527 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.992527 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:11.995073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.995073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:11.995073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:11.995073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:11.995073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.995073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:11.995073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:11.995073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.995073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:11.995073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:11.995073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:11.995073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.998344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:11.998344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:11.998344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:11.998344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:11.998344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:11.998344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:11.998344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:11.998344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:11.998344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:11.998344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:11.998344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:11.998344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:11.998344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:11.998990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.998990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:11.998990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.998990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:11.998990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:11.998990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:11.998990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:11.998990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:11.998990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.998990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:11.998990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:11.998990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.998990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:11.998990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:11.998990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:11.999371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:11.999772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:11.999772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:11.999772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:11.999772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:11.999772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:11.999772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:12.012343 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:12.017122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:12.018210 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:12.019642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:12.112978 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:12.117068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:12.117068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:12.117068 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:12.117068 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:12.117068 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:12.117068 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:12.117068 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:12.117068 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:12.117068 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:12.117068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.096 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:12.117068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:12.117068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:12.117068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:12.117068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:12.117068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:12.120742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:12.120742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:12.120742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:12.120742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:12.120742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:12.120742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:12.120742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:12.120742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:12.120742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:12.120742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:12.120742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:12.120742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:12.120742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:12.120742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:12.121642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:12.121642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:12.121642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:12.121642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:12.121642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:12.121642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:12.121642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:12.121642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:12.121642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:12.121642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:12.121642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:12.121642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:12.121642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:12.123936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:12.124360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:12.124360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:12.124360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:12.124360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:12.124360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:12.124360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:12.124360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:12.124360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:12.124360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:12.124360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:12.124360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:12.124360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:16.876843 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:16.883201 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:16.883201 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:16.883201 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:16.883201 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:16.883201 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:16.884381 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:16.884381 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:16.884381 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:16.884381 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:16.884381 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:16.890652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-741a0ca0-b62e-469e-b080-ea35c1e80ef1, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:16.890652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-044e825b-f1ca-4149-8ffc-6085a8ae6e1a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:16.890652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:16.890652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:16.890652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.890652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:16.890652 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:16.890652 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:16.890652 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:16.892222 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:16.892222 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:16.892222 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.892222 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:16.892222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.892222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:16.892222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:16.892222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:16.892222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.892222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:16.892222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:16.892222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.892222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:16.893060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:16.893060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:16.893060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.893060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:16.893060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:16.893060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:16.893060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:16.893060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:16.893060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:16.893060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:16.893060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:16.893060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:16.893060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:16.893060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:16.893654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:16.893654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:16.893654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.893654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:16.893654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.893654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:16.893654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:16.893654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:16.893654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:16.893654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:16.893654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.893654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:16.893654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:16.893654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:16.894116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:16.894644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:16.894644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:16.894644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:16.894644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:16.894644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:16.894644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:16.894644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:16.894644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.894644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:16.940277 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:16.940682 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:16.940941 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:16.941178 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:39:16.941847 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:16.942051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.944138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:16.945111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:16.945111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:16.945111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:16.945111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:16.945111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:16.945111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:16.945111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:16.945111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:16.945111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:16.945111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:16.945111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:16.945111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:16.945111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.945111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:16.946012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.946012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:16.946012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:16.946012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:16.946012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.946012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:16.946012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:16.946012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:39:16.946012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:16.946012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:16.946012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.946012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:16.946751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:16.946751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:16.946751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.946751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:16.946751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:16.946751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:16.946751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:16.946751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:16.946751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:16.946751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:16.946751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:16.946751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:16.946751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:16.946751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:16.947111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:16.947111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:16.947111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.947111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:16.948273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.948273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:16.948273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:16.948273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:16.948273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:16.948273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:16.948273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.948273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:16.948273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:16.948273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.948273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:16.948273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:16.948273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.948273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:16.948273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:16.948273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:16.948763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:16.949313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:16.949313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:16.949313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:16.949313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:16.949313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:16.949313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.949313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:16.951835 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:16.952200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:16.952402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:16.952633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:16.953212 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:16.953534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.954168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:16.954168 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:16.954168 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:16.954168 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:16.954168 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:16.954168 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:16.954168 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.954168 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:16.955145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.955145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:16.955145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:16.955145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:16.955145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.955145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:16.955145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:16.955145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:16.955145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.955145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:16.955145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:16.955145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:16.955145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:16.955145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:16.955145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:16.955428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:16.955428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:16.955428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:16.955428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:16.955428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:16.955428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:16.955428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:16.955428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.955428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:16.958563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.958563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:16.958563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:16.958563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:16.958563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.958563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:16.958563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:16.958563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:16.958563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:16.958563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.958563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:16.959308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:16.959308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:16.959308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.959308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:16.959308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:16.959308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:16.959308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:16.959308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:16.959308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:16.959308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:16.959308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:16.959308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:16.959308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:16.959308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:16.959600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:16.959600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:16.959600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.959600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.961817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:16.962214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:16.962626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:16.962626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:16.962626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.962626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:16.991313 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:16.991756 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:16.991922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:16.992149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:16.992558 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:16.992828 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:16.993182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.993754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:16.993754 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:16.993754 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:16.993754 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:16.993754 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:16.993754 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:16.993754 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.993754 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:16.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:16.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:16.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:16.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:16.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:16.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:16.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:16.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:16.994631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.994965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:16.994965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:16.994965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:16.994965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:16.994965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:16.994965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:16.994965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:16.994965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:16.994965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:16.994965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:16.994965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:16.994965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:16.994965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:16.995222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.995222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:16.995568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.995568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:16.995568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:16.995568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:16.995568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:16.995568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:16.995568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.995568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:16.995568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:16.995568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.995568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:16.995568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:16.995568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.995568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:16.995568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:16.995568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:16.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:16.996853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:16.996853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:16.996853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:17.022937 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:17.023213 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:17.023367 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:17.024366 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:17.087200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:17.087863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:17.088131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:17.088131 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:17.088131 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:17.088131 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:17.088131 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:17.088131 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:17.088131 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:17.088131 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:17.089314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.065 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:17.089314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:17.089314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:17.089314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:17.089314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:17.089314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:17.089314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:17.089314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:17.089314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:17.089314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:17.089314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:17.089314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:17.090521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:17.090521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:17.090521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:17.090521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:17.090521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:17.090521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:17.090521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:17.090521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:17.090521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:17.090521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:17.090521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:17.090521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:17.090521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:17.090833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:17.090833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:17.090833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:17.090833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:17.090833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:17.090833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:17.090833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:17.090833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:17.090833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:17.090833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:17.090833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:17.090833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:17.090833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:17.090833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:17.090833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:17.091215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:17.091702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:17.091702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:17.091702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:17.091702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:17.091702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:17.091702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:21.878183 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:21.882363 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:21.882363 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:21.882363 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:21.882363 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:21.882363 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:21.883317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:21.883317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:21.883317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:21.883317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:21.883317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:21.885963 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-044e825b-f1ca-4149-8ffc-6085a8ae6e1a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:21.885963 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:21.885963 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:21.885963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.885963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:21.885963 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:21.885963 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:21.885963 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:21.885963 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:21.885963 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:21.885963 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.885963 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:21.887828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.887828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:21.887828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:21.887828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:21.887828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.887828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:21.887828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:21.887828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.887828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:21.887828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:21.887828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:21.887828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.889596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:21.889596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:21.889596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:21.889596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:21.889596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:21.889596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:21.889596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:21.889596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:21.889596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:21.889596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:21.889596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:21.889596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:21.889596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:21.890086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.890086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:21.890086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.890086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:21.890086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:21.890086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:21.890086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:21.890086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:21.890086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.890086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:21.890086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:21.890086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.890086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:21.890086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:21.890086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:21.890519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:21.891036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:21.891036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:21.891036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:21.891036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:21.891036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.891036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:21.940937 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:21.941268 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:21.941389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:21.941546 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:39:21.942078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:21.942273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.942904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:21.942904 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:21.942904 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:21.942904 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:21.942904 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:21.942904 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:21.942904 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.942904 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:21.943972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.943972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:21.943972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:21.943972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:21.943972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.943972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:21.943972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:21.943972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:21.943972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.943972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:21.943972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:21.943972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:21.943972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:21.943972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:21.943972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:21.945166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:21.945166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:21.945166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:21.945166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:21.945166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:21.945166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:21.945166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:21.945166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.945166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:21.945166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.945166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:21.945166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:21.945166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:21.945166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.945954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:21.945954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:21.945954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:39:21.945954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:21.945954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:21.945954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.945954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:21.945954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:21.945954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:21.945954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.945954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:21.945954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:21.945954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:21.947596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:21.947596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:21.947596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:21.947596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:21.947596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:21.947596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:21.947596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:21.947596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:21.947596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:21.947596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:21.947596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.947596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:21.947831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.947831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:21.947831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:21.947831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:21.947831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:21.947831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:21.947831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.947831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:21.947831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:21.947831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.947831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:21.947831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:21.947831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.947831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:21.947831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:21.947831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:21.948128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:21.948679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:21.948679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:21.948679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:21.948679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:21.948679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:21.948679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.948679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:21.953755 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:21.954024 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:21.954143 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:21.954281 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:21.954641 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:21.954833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.955237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:21.955237 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:21.955237 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:21.955237 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:21.955237 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:21.955237 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:21.955237 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.955237 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:21.956040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.956040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:21.956040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:21.956040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:21.956040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.956040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:21.956040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:21.956040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:21.956040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.956040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:21.956040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:21.956040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:21.956040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:21.956040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:21.956040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:21.956306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:21.956306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:21.956306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:21.956306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:21.956306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:21.956306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:21.956306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:21.956306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.956306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:21.956941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.956941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:21.956941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:21.956941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:21.956941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.956941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:21.956941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:21.956941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:21.956941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:21.956941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.956941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:21.957636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:21.957636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:21.957636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.957636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:21.957636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:21.957636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:21.957636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:21.957636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:21.957636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:21.957636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:21.957636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:21.957636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:21.957636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:21.957636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:21.957911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:21.957911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:21.957911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.957911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.958046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:21.958351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:21.958658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:21.958658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:21.958658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.958658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:21.992915 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:21.993385 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:21.993608 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:21.993812 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:21.994235 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:21.994367 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:21.994671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.995159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:21.995159 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:21.995159 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:21.995159 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:21.995159 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:21.995159 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:21.995159 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.995159 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:21.996353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.996353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:21.996353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:21.996353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:21.996353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.996353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:21.996353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:21.996353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.996353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:21.996353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:21.996353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:21.996353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.996706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:21.996706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:21.996706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:21.996706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:21.996706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:21.996706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:21.996706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:21.996706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:21.996706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:21.996706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:21.996706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:21.996706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:21.996706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:21.996965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.996965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:21.997524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.997524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:21.997524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:21.997524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:21.997524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:21.997524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:21.997524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.997524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:21.997524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:21.997524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.997524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:21.997524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:21.997524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.997524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:21.997524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:21.997524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:21.997996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:21.998355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:21.998355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:21.998355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:22.023260 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:22.023648 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:22.023745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:22.024402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:22.025592 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=990,neutron:UPDATE=2,neutron:DELETE=14 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:39:22.064095 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:22.064680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:22.065369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:22.065369 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:22.065369 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:22.065369 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:22.065369 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:22.065369 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:22.065369 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:22.065369 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:22.066198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:22.066198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:22.066198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:22.066198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:22.066198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:22.066198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:22.066198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:22.066198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:22.066198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:22.066198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:22.066198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:22.066198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:22.066510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:22.066510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:22.066510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:22.066510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:22.066510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:22.066510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:22.066510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:22.066510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:22.066510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:22.066510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:22.066510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:22.066510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:22.066510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:22.066829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:22.066829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:22.067117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:22.067117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:22.067117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:22.067117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:22.067117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:22.067117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:22.067117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:22.067117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:22.067117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:22.067117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:22.067117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:22.067117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:22.067117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:22.067117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:22.067117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:22.067117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:22.067565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:22.067867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:22.067867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:22.067867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:26.878858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:26.883108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:26.883108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:26.883108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:26.883108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:26.883108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:26.885047 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:26.885047 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:26.885047 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:26.885047 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:26.885047 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:26.888599 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:26.888599 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:26.888599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.888599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:26.888599 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:26.888599 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:26.888599 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:26.888599 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:26.888599 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:26.888599 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.888599 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:26.888599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.891324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:26.891324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:26.891324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:26.891324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.891324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:26.891324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:26.891324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.891324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:26.891324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:26.891324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:26.891324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.891324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:26.891324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:26.891324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:26.892280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:26.892280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:26.892280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:26.892280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:26.892280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:26.892280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:26.892280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:26.892280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:26.892280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:26.892280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:26.892280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.892280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:26.892801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.892801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:26.892801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:26.892801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:26.892801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:26.892801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:26.892801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.892801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:26.892801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:26.892801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.892801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:26.892801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:26.892801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.892801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:26.892801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:26.892801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:26.893286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:26.893969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:26.893969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.893969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:26.941354 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:26.941603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:26.942383 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:26.942383 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:39:26.942383 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:26.942383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.942692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:26.942692 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:26.942692 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:26.942692 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:26.942692 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:26.942692 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:26.942692 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.942692 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:26.943872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.943872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:26.943872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:26.943872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:26.943872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.943872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:26.943872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:26.943872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:26.943872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.943872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:26.943872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:26.943872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:26.943872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:26.943872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:26.943872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:26.944697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:26.944697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:26.944697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:26.944697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:26.944697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:26.944697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:26.944697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:26.944697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.944697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:26.944697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.944697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:26.944697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:26.944697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:26.944697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.944990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:26.944990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:26.944990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:39:26.944990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:26.944990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:26.944990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.944990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:26.944990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:26.944990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:26.944990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.944990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:26.944990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:26.944990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:26.945221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:26.945221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:26.945221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:26.945221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:26.945221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:26.945221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:26.945221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:26.945221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:26.945221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:26.945221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:26.945221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.945221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:26.945428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.945428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:26.945428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:26.945428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:26.945428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:26.945428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:26.945428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.945428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:26.945428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:26.945428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.945428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:26.945428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:26.945428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.945428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:26.945428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:26.945428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:26.945737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:26.946395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:26.946395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:26.946395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:26.946395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:26.946395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:26.946395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.946395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:26.954076 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:26.954331 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:26.956793 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:26.957028 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:26.957465 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:26.957763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.958421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:26.958421 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:26.958421 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:26.958421 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:26.958421 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:26.958421 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:26.958421 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.958421 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:26.959269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.959269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:26.959269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:26.959269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:26.959269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.959269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:26.959269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:26.959269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:26.959269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.959269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:26.959269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:26.959269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:26.959269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:26.959269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:26.959269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:26.959767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:26.959767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:26.959767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:26.959767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:26.959767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:26.959767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:26.959767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:26.959767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.959767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:26.960023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.960023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:26.960023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:26.960023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:26.960023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.960023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:26.960023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:26.960023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:26.960023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:26.960023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.960023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:26.961332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:26.961332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:26.961332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.961332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:26.961332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:26.961332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:26.961332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:26.961332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:26.961332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:26.961332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:26.961332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:26.961332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:26.961332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:26.961332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:26.961762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:26.961762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:26.961762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.961762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:26.961762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.961762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:26.961762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:26.961762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:26.961762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:26.961762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:26.961762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.961762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:26.961762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:26.961762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:26.962167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:26.962717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:26.962717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:26.962717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:26.962717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:26.962717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:26.962717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:26.962717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:26.962717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:26.962717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:26.962717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.962717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:26.993454 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:26.993981 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:26.994102 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:26.994237 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:26.994733 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:26.994969 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:26.995205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.995763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:26.995763 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:26.995763 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:26.995763 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:26.995763 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:26.995763 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:26.995763 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.995763 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:26.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:26.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:26.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:26.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:26.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:26.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:26.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:26.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:26.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.997089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:26.997089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:26.997089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:26.997089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:26.997089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:26.997089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:26.997089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:26.997089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:26.997089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:26.997089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:26.997089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:26.997089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:26.997089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:26.997370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.997370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:26.998675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:26.999006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:26.999006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:26.999006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:27.023706 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:27.024169 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:27.024265 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:27.024915 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:27.064897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:27.065070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:27.065689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:27.065689 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:27.065689 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:27.065689 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:27.065689 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:27.065689 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:27.065689 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:27.065689 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:27.066731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:27.066731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:27.066731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:27.066731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:27.066731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:27.066731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:27.066731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:27.066731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:27.066731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:27.066731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:27.066731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:27.066731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:27.066983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:27.066983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:27.066983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:27.066983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:27.066983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:27.066983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:27.066983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:27.066983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:27.066983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:27.066983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:27.066983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:27.066983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:27.066983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:27.067181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:27.067181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:27.067452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:27.067452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:27.067452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:27.067452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:27.067452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:27.067452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:27.067452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:27.067452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:27.067452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:27.067452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:27.067452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:27.067452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:27.067452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:27.067452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:27.067452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:27.067452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:27.067792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:27.068088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:27.068088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:27.068088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:31.880357 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:31.880959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:31.880959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:31.881847 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:31.882065 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:31.882215 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:31.882332 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:31.882492 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:31.882638 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:31.882758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:31.882865 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:31.883003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:31.883109 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:31.883215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.883730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:31.883730 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:31.883730 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:31.883730 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:31.883730 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:31.883730 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:31.883730 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.883730 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:31.884619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.884619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:31.884619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:31.884619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:31.884619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.884619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:31.884619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:31.884619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.884619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:31.884619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:31.884619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:31.884619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.884901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:31.884901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:31.884901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:31.884901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:31.884901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:31.884901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:31.884901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:31.884901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:31.884901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:31.884901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:31.884901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:31.884901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:31.884901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:31.885211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.885211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:31.885343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.885343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:31.885343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:31.885343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:31.885343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:31.885343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:31.885343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.885343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:31.885343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:31.885343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.885343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:31.885343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:31.885343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.885343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:31.885343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:31.885343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:31.885746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:31.886237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:31.886237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.886237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:31.944185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:31.944185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:31.944185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:31.944185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:39:31.944185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:31.944185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.944378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:31.944714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:31.944714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:31.944714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:31.944714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:31.944714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:31.944714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:31.944714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:31.944714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:31.944714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:31.944714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:31.944714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:31.944714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:31.944714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.944714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:31.944945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.944945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:31.944945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:31.944945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:31.944945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.944945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:31.944945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:31.944945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:39:31.944945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:31.944945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:31.944945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.944945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:31.945171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:31.945171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:31.945171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.945171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:31.945171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:31.945171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:31.945171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:31.945171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:31.945171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:31.945171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:31.945171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:31.945171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:31.945171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:31.945171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:31.945414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:31.945414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:31.945414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.945414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:31.945577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.945577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:31.945577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:31.945577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:31.945577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:31.945577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:31.945577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.945577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:31.945577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:31.945577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.945577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:31.945577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:31.945577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.945577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:31.945577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:31.945577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:31.945867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:31.946139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:31.946139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:31.946139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:31.946139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:31.946139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:31.946139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.946139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:31.954482 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:31.954819 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:31.954955 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:31.955106 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:31.955637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:31.955839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.956256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:31.956256 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:31.956256 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:31.956256 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:31.956256 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:31.956256 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:31.956256 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.956256 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:31.957071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.957071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:31.957071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:31.957071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:31.957071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.957071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:31.957071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:31.957071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:31.957071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.957071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:31.957071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:31.957071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:31.957071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:31.957071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:31.957071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:31.957321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:31.957321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:31.957321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:31.957321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:31.957321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:31.957321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:31.957321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:31.957321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.957321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:31.957486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.957486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:31.957486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:31.957486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:31.957486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.957486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:31.957486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:31.957486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:31.957486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:31.957486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.957486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:31.957684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:31.957684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:31.957684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.957684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:31.957684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:31.957684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:31.957684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:31.957684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:31.957684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:31.957684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:31.957684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:31.957684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:31.957684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:31.957684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:31.957891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:31.957891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:31.957891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.957891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.958172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:31.958441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:31.958770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:31.958770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:31.958770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.958770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:31.994213 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:31.994522 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:31.994741 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:31.994882 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:31.995437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:31.995749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:31.996118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.996908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:31.996908 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:31.996908 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:31.996908 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:31.996908 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:31.996908 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:31.996908 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.996908 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:31.998298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.998298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:31.998298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:31.998298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:31.998298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.998298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:31.998298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:31.998298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.998298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:31.998298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:31.998298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:31.998298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.998598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:31.998598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:31.998598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:31.998598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:31.998598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:31.998598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:31.998598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:31.998598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:31.998598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:31.998598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:31.998598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:31.998598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:31.998598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:31.998868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.998868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:31.998960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.998960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:31.998960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:31.998960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:31.998960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:31.998960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:31.998960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.998960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:31.998960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:31.998960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.998960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:31.998960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:31.998960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.998960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:31.998960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:31.998960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:31.999286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:31.999761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:31.999761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:31.999761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:32.024268 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:32.024710 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:32.024920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:32.025418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:32.063769 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:32.064002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:32.064644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:32.064644 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:32.064644 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:32.064644 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:32.064644 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:32.064644 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:32.064644 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:32.064644 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:32.065579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:32.065579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:32.065579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:32.065579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:32.065579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:32.065579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:32.065579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:32.065579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:32.065579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:32.065579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:32.065579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:32.065579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:32.065837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:32.065837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:32.065837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:32.065837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:32.065837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:32.065837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:32.065837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:32.065837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:32.065837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:32.065837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:32.065837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:32.065837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:32.065837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:32.066084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:32.066084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:32.066548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:32.066548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:32.066548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:32.066548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:32.066548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:32.066548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:32.066548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:32.066548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:32.066548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:32.066548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:32.066548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:32.066548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:32.066548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:32.066548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:32.066548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:32.066548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:32.067015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:32.067550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:32.067550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:32.067550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:36.880914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:36.881519 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:36.881519 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:36.882796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:36.883286 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:36.883286 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:36.883286 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:36.883450 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:36.883792 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:36.884087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:36.884453 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:36.884453 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:36.884764 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:36.884914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.885712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:36.885712 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:36.885712 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:36.885712 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:36.885712 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:36.885712 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:36.885712 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.885712 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:36.886766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.886766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:36.886766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:36.886766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:36.886766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.886766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:36.886766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:36.886766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.886766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:36.886766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:36.886766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:36.886766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.887294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:36.887294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:36.887294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:36.887294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:36.887294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:36.887294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:36.887294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:36.887294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:36.887294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:36.887294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:36.887294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:36.887294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:36.887294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:36.887737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.887737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:36.887737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.887737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:36.887737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:36.887737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:36.887737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:36.887737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:36.887737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.887737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:36.887737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:36.887737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.887737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:36.887737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:36.887737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:36.888223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:36.888913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:36.888913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:36.888913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:36.888913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:36.888913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.888913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:36.942241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:36.942895 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:36.942895 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:36.942895 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:39:36.943402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:36.943610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.944652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:36.944652 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:36.944652 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:36.944652 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:36.944652 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:36.944652 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:36.944652 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.944652 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:36.945419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.945419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:36.945419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:36.945419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:36.945419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.945419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:36.945419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:36.945419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:36.945419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.945419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:36.945419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:36.945419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:36.945419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:36.945419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:36.945419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:36.945970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:36.945970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:36.945970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:36.945970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:36.945970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:36.945970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:36.945970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:36.945970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.945970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:36.946291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.946291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:36.946291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:36.946291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:36.946291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.946291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:36.946291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:36.946291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:39:36.946291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:36.946291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:36.946291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.946291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:36.946727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:36.946727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:36.946727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.946727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:36.946727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:36.946727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:36.946727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:36.946727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:36.946727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:36.946727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:36.946727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:36.946727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:36.946727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:36.946727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:36.947184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:36.947184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:36.947184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.947184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:36.947320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.947320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:36.947320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:36.947320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:36.947320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:36.947320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:36.947320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.947320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:36.947320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:36.947320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.947320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:36.947320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:36.947320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.947320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:36.947320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:36.947320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:36.947715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:36.948069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:36.948069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:36.948069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:36.948069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:36.948069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:36.948069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.948069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:36.955103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:36.955342 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:36.955387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:36.955509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:36.955850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:36.956064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.956429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:36.956429 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:36.956429 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:36.956429 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:36.956429 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:36.956429 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:36.956429 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.956429 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:36.957055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.957055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:36.957055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:36.957055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:36.957055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.957055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:36.957055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:36.957055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:36.957055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.957055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:36.957055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:36.957055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:36.957055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:36.957055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:36.957055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:36.957309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:36.957309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:36.957309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:36.957309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:36.957309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:36.957309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:36.957309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:36.957309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.957309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:36.957668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.957668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:36.957668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:36.957668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:36.957668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.957668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:36.957668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:36.957668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:36.957668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:36.957668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.957668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:36.957892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:36.957892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:36.957892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.957892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:36.957892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:36.957892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:36.957892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:36.957892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:36.957892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:36.957892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:36.957892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:36.957892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:36.957892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:36.957892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:36.958104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:36.958104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:36.958104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.958104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.958418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:36.958893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:36.959192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:36.959192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:36.959192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.959192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:36.994801 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:36.995179 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:36.995310 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:36.995494 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:36.995801 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:36.995992 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:36.996229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.996661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:36.996661 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:36.996661 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:36.996661 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:36.996661 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:36.996661 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:36.996661 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.996661 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:36.997348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.997348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:36.997348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:36.997348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:36.997348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.997348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:36.997348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:36.997348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.997348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:36.997348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:36.997348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:36.997348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.998175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:36.998175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:36.998175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:36.998175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:36.998175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:36.998175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:36.998175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:36.998175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:36.998175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:36.998175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:36.998175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:36.998175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:36.998175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:36.998565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.998565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:36.998565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.998565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:36.998565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:36.998565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:36.998565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:36.998565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:36.998565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.998565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:36.998565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:36.998565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.998565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:36.998565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:36.998565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:36.998928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:36.999219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:36.999219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:36.999219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:36.999219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:36.999219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:36.999219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:37.024818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:37.025097 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:37.025188 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:37.025854 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:37.071942 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:37.072179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:37.072817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:37.072817 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:37.072817 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:37.072817 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:37.072817 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:37.072817 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:37.072817 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:37.072817 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:37.073672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:37.073672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:37.073672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:37.073672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:37.073672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:37.073672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:37.073672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:37.073672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:37.073672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:37.073672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:37.073672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:37.073672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:37.073976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:37.073976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:37.073976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:37.073976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:37.073976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:37.073976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:37.073976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:37.073976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:37.073976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:37.073976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:37.073976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:37.073976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:37.073976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:37.074239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:37.074239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:37.074363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:37.074363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:37.074363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:37.074363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:37.074363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:37.074363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:37.074363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:37.074363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:37.074363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:37.074363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:37.074363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:37.074363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:37.074363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:37.074363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:37.074363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:37.074363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:37.074718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:37.075106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:37.075106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:37.075106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:41.881402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:41.882067 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:41.882067 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:41.884816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:41.884816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:41.884816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:41.884816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:41.884816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:41.885023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:41.885023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:41.885023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:41.885023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:41.885023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:41.885615 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:41.885615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.885615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:41.885615 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:41.885615 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:41.885615 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:41.885615 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:41.885615 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:41.885615 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.885615 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:41.886018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.886018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:41.886018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:41.886018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:41.886018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:41.886018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:41.886018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:41.886018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:41.886018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:41.886018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:41.886018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:41.886018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:41.886317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:41.886317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:41.886317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:41.886317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:41.886317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:41.886317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:41.886317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:41.886317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:41.886317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:41.886317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:41.886317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:41.886317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:41.886317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:41.886613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.886613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:41.887341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.887341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:41.887341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:41.887341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:41.887341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:41.887341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:41.887341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:41.887341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:41.887341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:41.887341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:41.887341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:41.887341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:41.887341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:41.887341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:41.887341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:41.887341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:41.887757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:41.888157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:41.888157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.888157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:41.942695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:41.943061 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:41.943162 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:41.943266 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:39:41.944172 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:41.944172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.944355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:41.944355 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:41.944355 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:41.944355 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:41.944355 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:41.944355 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:41.944355 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.944355 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:41.945291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.945291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:41.945291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:41.945291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:41.945291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:41.945291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:41.945291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:41.945291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:41.945291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:41.945291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:41.945291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:41.945291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:41.945291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:41.945291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:41.945291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:41.945761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:41.945761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:41.945761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:41.945761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:41.945761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:41.945761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:41.945761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:41.945761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.945761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:41.946236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.946236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:41.946236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:41.946236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:41.946236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:41.946236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:41.946236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:41.946236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:39:41.946236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:41.946236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:41.946236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:41.946236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:41.946650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:41.946650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:41.946650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:41.946650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:41.946650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:41.946650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:41.946650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:41.946650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:41.946650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:41.946650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:41.946650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:41.946650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:41.946650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:41.946650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:41.947012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:41.947012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:41.947012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.947012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:41.947225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.947225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:41.947225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:41.947225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:41.947225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:41.947225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:41.947225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:41.947225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:41.947225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:41.947225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:41.947225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:41.947225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:41.947225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:41.947225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:41.947225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:41.947225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:41.947689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:41.948158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:41.948158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:41.948158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:41.948158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:41.948158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:41.948158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.948158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:41.955426 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:41.955756 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:41.955898 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:41.956055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:41.958490 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:41.958490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.958490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:41.958490 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:41.958490 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:41.958490 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:41.958490 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:41.958490 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:41.958490 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.958490 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:41.958490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.958490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:41.958490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:41.958490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:41.959003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:41.959438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.959438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:41.959438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.959438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:41.959438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:41.959438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:41.959438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:41.959438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:41.959438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:41.959438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:41.959438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:41.959438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:41.959849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:41.959849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:41.959849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:41.959849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:41.959849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:41.959849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:41.959849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:41.959849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:41.959849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:41.959849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:41.959849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:41.959849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:41.959849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:41.959849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:41.960196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:41.960196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:41.960196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:41.960196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.960196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:41.960196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.960196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:41.960196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:41.960196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:41.960196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:41.960196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:41.960196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:41.960196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:41.960196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:41.960196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:41.962787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:41.963402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:41.963402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:41.963402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:41.963402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:41.963402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:41.963402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:41.963402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:41.963402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:41.963402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:41.963402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.963402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:41.995918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:41.995918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:41.995918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:41.995918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:41.998490 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:41.998490 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:41.998490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.999430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:41.999430 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:41.999430 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:41.999430 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:41.999430 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:41.999430 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:41.999430 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:41.999430 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:42.000152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:42.000152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:42.000152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:42.000152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:42.000152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:42.000152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:42.000152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:42.000152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:42.000152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:42.000152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:42.000152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:42.000152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:42.000565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:42.000565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:42.000565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:42.000565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:42.000565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:42.000565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:42.000565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:42.000565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:42.000565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:42.000565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:42.000565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:42.000565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:42.000565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:42.000954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:42.000954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:42.000954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:42.000954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:42.000954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:42.000954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:42.000954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:42.000954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:42.000954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:42.000954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:42.000954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:42.000954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:42.000954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:42.000954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:42.000954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:42.001402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:42.002022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:42.002022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:42.002022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:42.002022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:42.002022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:42.002022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:42.025190 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:42.025559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:42.025559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:42.026158 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:42.074290 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:42.074597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:42.075309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:42.075309 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:42.075309 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:42.075309 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:42.075309 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:42.075309 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:42.075309 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:42.075309 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:42.076436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.050 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:42.076436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:42.076436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:42.076436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:42.076436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:42.076436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:42.076436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:42.076436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:42.076436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:42.076436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:42.076436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:42.076436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:42.078243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:42.078243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:42.078243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:42.078243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:42.078243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:42.078243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:42.078243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:42.078243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:42.078243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:42.078243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:42.078243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:42.078243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:42.078243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:42.078761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:42.078761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:42.078761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:42.078761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:42.078761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:42.078761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:42.078761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:42.078761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:42.078761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:42.078761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:42.078761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:42.078761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:42.078761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:42.078761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:42.078761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:42.079198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:42.079731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:42.079731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:42.079731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:42.079731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:42.079731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:42.079731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:46.882847 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:46.884202 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:46.884302 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:46.885684 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:46.887589 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:46.887753 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:46.887898 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:46.888042 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:46.888233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:46.888401 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:46.888724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:46.889082 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:46.889268 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:46.889459 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:46.889674 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:46.889976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.892391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:46.892391 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:46.892391 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:46.892391 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:46.892391 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:46.892391 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:46.892391 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.892391 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:46.893142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.893142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:46.893142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:46.893142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:46.893142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:46.893142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:46.893142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:46.893142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:46.893142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:46.893142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:46.893142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:46.893142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:46.893608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:46.893608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:46.893608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:46.893608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:46.893608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:46.893608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:46.893608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:46.893608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:46.893608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:46.893608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:46.893608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:46.893608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:46.893608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:46.894052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.894052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:46.894250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.894250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:46.894250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:46.894250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:46.894250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:46.894250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:46.894250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:46.894250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:46.894250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:46.894250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:46.894250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:46.894250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:46.894250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:46.894250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:46.894250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:46.894250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:46.894827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:46.895369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:46.895369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.895369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:46.943287 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:46.943618 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:46.943715 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:46.943830 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:39:46.944304 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:46.944500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.944986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:46.944986 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:46.944986 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:46.944986 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:46.944986 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:46.944986 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:46.944986 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.944986 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:46.945607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.945607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:46.945607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:46.945607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:46.945607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:46.945607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:46.945607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:46.945607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:46.945607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:46.945607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:46.945607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:46.945607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:46.945607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:46.945607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:46.945607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:46.945990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:46.945990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:46.945990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:46.945990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:46.945990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:46.945990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:46.945990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:46.945990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.945990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:46.946254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.946254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:46.946254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:46.946254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:46.946254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:46.946254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:46.946254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:46.946254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:39:46.946254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:46.946254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:46.946254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:46.946254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:46.946505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:46.946505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:46.946505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:46.946505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:46.946505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:46.946505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:46.946505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:46.946505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:46.946505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:46.946505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:46.946505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:46.946505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:46.946505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:46.946505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:46.946767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:46.946767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:46.946767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.946767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:46.947383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.947383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:46.947383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:46.947383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:46.947383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:46.947383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:46.947383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:46.947383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:46.947383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:46.947383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:46.947383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:46.947383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:46.947383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:46.947383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:46.947383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:46.947383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:46.947740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:46.948189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:46.948189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:46.948189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:46.948189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:46.948189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:46.948189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.948189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:46.955929 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:46.956923 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:46.957084 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:46.957239 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:46.957691 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:46.957895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.958430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:46.958430 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:46.958430 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:46.958430 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:46.958430 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:46.958430 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:46.958430 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.958430 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:46.959262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.959262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:46.959262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:46.959262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:46.959262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:46.959262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:46.959262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:46.959262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:46.959262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:46.959262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:46.959262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:46.959262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:46.959262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:46.959262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:46.959262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:46.959673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:46.959673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:46.959673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:46.959673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:46.959673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:46.959673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:46.959673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:46.959673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.959673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:46.960264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.960264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:46.960264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:46.960264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:46.960264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:46.960264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:46.960264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:46.960264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:46.960264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:46.960264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:46.960264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:46.960519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:46.960519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:46.960519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:46.960519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:46.960519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:46.960519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:46.960519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:46.960519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:46.960519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:46.960519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:46.960519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:46.960519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:46.960519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:46.960519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:46.960873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:46.960873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:46.960873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.960873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:46.961343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:46.961860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:46.962203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:46.962203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:46.962203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:46.962203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:46.995897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:46.998912 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:46.999050 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:46.999247 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:46.999695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:46.999961 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:47.000460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:47.001091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:47.001091 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:47.001091 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:47.001091 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:47.001091 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:47.001091 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:47.001091 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:47.001091 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:47.002135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:47.002135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:47.002135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:47.002135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:47.002135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:47.002135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:47.002135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:47.002135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:47.002135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:47.002135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:47.002135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:47.002135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:47.002500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:47.002500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:47.002500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:47.002500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:47.002500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:47.002500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:47.002500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:47.002500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:47.002500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:47.002500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:47.002500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:47.002500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:47.002500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:47.002871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:47.002871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:47.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:47.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:47.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:47.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:47.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:47.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:47.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:47.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:47.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:47.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:47.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:47.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:47.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:47.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:47.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:47.003505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:47.004001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:47.004598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:47.004598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:47.004598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:47.025708 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:47.025948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:47.026059 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:47.026801 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:47.082899 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:47.083257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:47.083982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:47.083982 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:47.083982 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:47.083982 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:47.083982 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:47.083982 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:47.083982 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:47.083982 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:47.085076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:47.085076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:47.085076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:47.085076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:47.085076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:47.085076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:47.085076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:47.085076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:47.085076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:47.085076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:47.085076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:47.085076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:47.085327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:47.085327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:47.085327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:47.085327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:47.085327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:47.085327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:47.085327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:47.085327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:47.085327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:47.085327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:47.085327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:47.085327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:47.085327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:47.085596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:47.085596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:47.086827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:47.086827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:47.086827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:47.086827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:47.086827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:47.086827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:47.086827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:47.086827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:47.086827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:47.086827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:47.086827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:47.086827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:47.086827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:47.086827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:47.086827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:47.086827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:47.087318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:47.087986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:47.087986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:47.087986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:51.884361 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:51.884975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:51.885136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:51.886659 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:51.887225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:51.887225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:51.887225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:51.887344 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-c4e30c20-6542-404e-bcea-c07f4273b5a8, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:51.887385 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:51.887508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:51.887674 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:51.887796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:51.887931 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:51.888038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:51.888161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-c1b28d00-499a-4ef4-8e9f-9de860050fa2, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:51.888277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.888874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:51.888874 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:51.888874 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:51.888874 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:51.888874 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:51.888874 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:51.888874 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.888874 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:51.890048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.890048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:51.890048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:51.890048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:51.890048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:51.890048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:51.890048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:51.890048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:51.890048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:51.890048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:51.890048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:51.890048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:51.890458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:51.890458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:51.890458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:51.890458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:51.890458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:51.890458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:51.890458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:51.890458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:51.890458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:51.890458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:51.890458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:51.890458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:51.890458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:51.890804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.890804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:51.891275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.891275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:51.891275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:51.891275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:51.891275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:51.891275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:51.891275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:51.891275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:51.891275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:51.891275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:51.891275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:51.891275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:51.891275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:51.891275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:51.891275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:51.891275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:51.891753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:51.892250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:51.892250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.892250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:51.943708 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:51.944681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:51.944681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:51.944681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:39:51.944984 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:51.945211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:51.948568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:51.949195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:51.949195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:51.949195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:51.949195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:51.949195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:51.949195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:51.949195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:51.949195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:51.949195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:51.949195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:51.949195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:51.949195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:51.949195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.949195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:51.949195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.949632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:51.949632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:51.949632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:51.949632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:51.949632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:51.949632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:51.949632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:39:51.949632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:51.949632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:51.949632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:51.949632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:51.949632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:51.949632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:51.949979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:51.949979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:51.949979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:51.949979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:51.949979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:51.949979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:51.949979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:51.949979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:51.949979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:51.949979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:51.949979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:51.949979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:51.949979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:51.949979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:51.950315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.950315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:51.950950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.950950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:51.950950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:51.950950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:51.950950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:51.950950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:51.950950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:51.950950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:51.950950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:51.950950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:51.950950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:51.950950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:51.950950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:51.950950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:51.950950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:51.950950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:51.951488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:51.952175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:51.952175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:51.952175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:51.952175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:51.952175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:51.952175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.952175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:51.956949 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:51.957592 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:51.957592 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:51.957592 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:51.957989 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:51.958306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:51.961260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:51.961885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:51.961885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:51.961885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:51.961885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:51.961885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:51.961885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:51.961885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:51.961885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:51.961885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:51.961885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:51.961885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:51.961885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:51.961885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.961885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:51.961885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.962369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:51.962369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:51.962369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:51.962369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:51.962369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:51.962369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:51.962369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:51.962369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:51.962369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:51.962369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:51.962369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:51.962369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:51.962369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:51.962786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:51.962786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:51.962786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:51.962786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:51.962786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:51.962786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:51.962786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:51.962786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:51.962786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:51.962786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:51.962786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:51.962786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:51.962786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:51.963106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.963106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:51.963106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.963106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:51.963106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:51.963106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:51.963106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:51.963106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:51.963106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:51.963106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:51.963106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:51.963106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:51.963106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:51.963106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:51.963106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:51.963520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:51.964037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:51.964037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:51.964037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:51.964037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:51.964037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:51.964037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:51.964037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:51.964037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:51.998944 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:51.999491 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:51.999491 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:51.999491 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:51.999751 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:51.999961 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:52.000293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:52.001746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:52.001746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:52.001746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:52.001746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:52.001746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:52.001746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:52.001746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:52.001746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:52.001746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:52.001746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:52.001746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:52.001746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:52.002020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:52.002020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:52.002020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:52.002020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:52.002020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:52.002020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:52.002020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:52.002020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:52.002020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:52.002020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:52.002020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:52.002020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:52.002020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:52.002289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:52.002289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:52.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:52.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:52.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:52.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:52.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:52.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:52.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:52.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:52.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:52.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:52.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:52.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:52.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:52.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:52.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:52.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:52.003947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:52.004280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:52.004280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:52.004280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:52.028358 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:52.028358 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:52.028358 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:52.028358 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:52.075449 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:52.075773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:52.076473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:52.076473 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:52.076473 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:52.076473 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:52.076473 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:52.076473 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:52.076473 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:52.076473 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:52.078563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.050 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:52.078563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:52.078563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:52.078563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:52.078563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:52.078563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:52.078563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:52.078563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:52.078563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:52.078563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:52.078563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:52.078563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:52.078895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:52.078895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:52.078895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:52.078895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:52.078895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:52.078895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:52.078895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:52.078895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:52.078895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:52.078895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:52.078895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:52.078895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:52.078895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:52.079197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:52.079197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:52.079197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:52.079197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:52.079197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:52.079197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:52.079197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:52.079197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:52.079197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:52.079197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:52.079197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:52.079197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:52.079197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:52.079197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:52.079197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:52.079511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:52.079908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:52.079908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:52.079908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:52.079908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:52.079908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:52.079908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:56.885006 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:56.885675 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:56.885675 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:56.886565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:56.886744 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:56.886992 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:56.887159 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:56.887301 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:56.887420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:56.887561 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:56.887713 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:56.887950 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-065f5984-bcd4-48bd-b8cb-3119f0d33351, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:56.888083 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:56.888227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.888865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:56.888865 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:56.888865 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:56.888865 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:56.888865 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:56.888865 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:56.888865 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.888865 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:56.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:56.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:56.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:56.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:56.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:56.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:56.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:56.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:56.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:56.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:56.889563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:56.890010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:56.890010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:56.890010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:56.890010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:56.890010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:56.890010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:56.890010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:56.890010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:56.890010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:56.890010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:56.890010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:56.890010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:56.890010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:56.890560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.890560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:56.890560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.890560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:56.890560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:56.890560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:56.890560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:56.890560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:56.890560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:56.890560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:56.890560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:56.890560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:56.890560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:56.890560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:56.890560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:56.891022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:56.891592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:56.891592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:56.891592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:56.891592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:56.891592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.891592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:56.944678 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:56.944678 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:56.944802 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:56.944871 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:39:56.945264 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:56.945442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.946058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:56.946058 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:56.946058 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:56.946058 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:56.946058 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:56.946058 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:56.946058 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.946058 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:56.946864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.946864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:56.946864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:56.946864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:56.946864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:56.946864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:56.946864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:56.946864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:56.946864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:56.946864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:56.946864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:56.946864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:56.946864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:56.946864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:56.946864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:56.947345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:56.947345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:56.947345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:56.947345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:56.947345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:56.947345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:56.947345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:56.947345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.947345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:56.947725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.947725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:56.947725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:56.947725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:56.947725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:56.947725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:56.947725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:56.947725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:39:56.947725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:56.947725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:56.947725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:56.947725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:56.947939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:56.947939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:56.947939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:56.947939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:56.947939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:56.947939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:56.947939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:56.947939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:56.947939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:56.947939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:56.947939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:56.947939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:56.947939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:56.947939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:56.948169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:56.948169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:56.948169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.948169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:56.949520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.949520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:56.949520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:56.949520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:56.949520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:56.949520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:56.949520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:56.949520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:56.949520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:56.949520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:56.949520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:56.949520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:56.949520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:56.949520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:56.949520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:56.949520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:56.949860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:56.950157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:56.950157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:56.950157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:56.950157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:56.950157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:56.950157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.950157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:56.957426 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:56.958784 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:56.958918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:56.959032 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:56.961844 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:56.962036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.962570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:56.962570 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:56.962570 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:56.962570 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:56.962570 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:56.962570 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:56.962570 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.962570 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:56.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:39:56.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:56.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:56.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:56.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:56.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:56.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:39:56.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:56.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:39:56.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:56.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:39:56.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:56.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:39:56.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:39:56.963387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:56.963387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:39:56.963387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:56.963387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:39:56.963387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:39:56.963387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:56.963387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:39:56.963387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.963387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:39:56.963774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.963774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:56.963774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:56.963774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:56.963774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:56.963774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:56.963774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:56.963774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:56.963774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:56.963774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:56.963774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:56.964033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:56.964033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:56.964033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:56.964033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:56.964033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:56.964033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:56.964033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:56.964033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:56.964033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:56.964033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:56.964033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:56.964033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:56.964033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:56.964033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:56.964268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:56.964268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:56.964268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.964268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:56.964600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:56.964939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:56.965277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:56.965277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:56.965277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:56.965277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:56.999914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:57.000300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:57.000483 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:57.000679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:57.000958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:57.001162 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:57.001387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:57.002418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:57.002418 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:57.002418 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:57.002418 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:57.002418 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:57.002418 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:57.002418 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:57.002418 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:57.003073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:57.003073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:57.003073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:57.003073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:57.003073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:57.003073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:57.003073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:57.003073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:57.003073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:57.003073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:57.003073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:57.003073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:57.003313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:57.003313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:57.003313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:57.003313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:57.003313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:57.003313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:57.003313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:57.003313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:57.003313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:57.003313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:57.003313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:57.003313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:57.003313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:57.003590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:57.003590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:57.003854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:57.003854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:57.003854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:57.003854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:57.003854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:57.003854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:57.003854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:57.003854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:57.003854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:57.003854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:57.003854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:57.003854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:57.003854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:57.003854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:57.003854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:57.003854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:57.004290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:57.004936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:57.004936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:57.004936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:39:57.027084 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:39:57.027596 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:39:57.027802 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:39:57.028708 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:39:57.096136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:39:57.096383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:57.097032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:39:57.097032 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:57.097032 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:39:57.097032 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:39:57.097032 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:57.097032 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:39:57.097032 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:57.097032 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:39:57.097913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.069 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:57.097913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:39:57.097913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:57.097913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:39:57.097913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:57.097913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:57.097913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:57.097913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:57.097913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:57.097913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:57.097913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:39:57.097913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:57.098167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:39:57.098167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:57.098167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:39:57.098167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:57.098167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:39:57.098167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:39:57.098167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:57.098167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:39:57.098167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:57.098167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:39:57.098167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:39:57.098167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:57.098167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:39:57.098596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:57.098596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:39:57.098917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:57.098917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:39:57.098917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:39:57.098917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:39:57.098917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:39:57.098917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:39:57.098917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:57.098917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:39:57.098917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:39:57.098917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:57.098917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:39:57.098917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:39:57.098917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:39:57.098917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:39:57.098917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:39:57.098917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:39:57.099219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:39:57.099608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:39:57.099608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:39:57.099608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:01.885630 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:01.886209 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:01.886209 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:01.887752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:01.887879 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:01.888198 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:01.888285 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:01.888494 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:01.888650 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:01.888771 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:01.888941 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:01.889347 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:01.889347 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:01.889347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.890045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:01.890045 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:01.890045 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:01.890045 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:01.890045 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:01.890045 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:01.890045 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.890045 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:01.891215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.891215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:01.891215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:01.891215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:01.891215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:01.891215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:01.891215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:01.891215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:01.891215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:01.891215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:01.891215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:01.891215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:01.891632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:01.891632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:01.891632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:01.891632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:01.891632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:01.891632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:01.891632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:01.891632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:01.891632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:01.891632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:01.891632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:01.891632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:01.891632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:01.892055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.892055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:01.892518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.892518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:01.892518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:01.892518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:01.892518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:01.892518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:01.892518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:01.892518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:01.892518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:01.892518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:01.892518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:01.892518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:01.892518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:01.892518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:01.892518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:01.892518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:01.892989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:01.895905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:01.895905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.895905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:01.944923 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:01.945563 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:01.945689 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:01.945837 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:40:01.946562 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:01.946815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.947377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:01.947377 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:01.947377 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:01.947377 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:01.947377 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:01.947377 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:01.947377 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.947377 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:01.948054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.948054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:01.948054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:01.948054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:01.948054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:01.948054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:01.948054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:01.948054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:01.948054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:01.948054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:01.948054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:01.948054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:01.948054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:01.948054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:01.948054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:01.948584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:01.948584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:01.948584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:01.948584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:01.948584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:01.948584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:01.948584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:01.948584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.948584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:01.948883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.948883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:01.948883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:01.948883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:01.948883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:01.948883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:01.948883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:01.948883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:40:01.948883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:01.948883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:01.948883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:01.948883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:01.949255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:01.949255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:01.949255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:01.949255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:01.949255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:01.949255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:01.949255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:01.949255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:01.949255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:01.949255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:01.949255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:01.949255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:01.949255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:01.949255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:01.949698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:01.949698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:01.949698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.949698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:01.949698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.949698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:01.949698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:01.949698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:01.949698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:01.949698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:01.949698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:01.949698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:01.949698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:01.949698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:01.950177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:01.950774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:01.950774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:01.950774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:01.950774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:01.950774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:01.950774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:01.950774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:01.950774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:01.950774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:01.950774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:01.950774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.950774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:01.958960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:01.959194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:01.959307 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:01.959476 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:01.960745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:01.961020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.961800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:01.961800 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:01.961800 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:01.961800 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:01.961800 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:01.961800 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:01.961800 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.961800 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:01.962763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.962763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:01.962763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:01.962763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:01.962763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:01.962763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:01.962763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:01.962763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:01.962763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:01.962763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:01.962763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:01.962763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:01.962763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:01.962763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:01.962763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:01.963718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:01.963718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:01.963718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:01.963718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:01.963718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:01.963718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:01.963718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:01.963718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.963718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:01.964001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.964001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:01.964001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:01.964001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:01.964001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:01.964001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:01.964001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:01.964001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:01.964001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:01.964001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:01.964001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:01.964395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:01.964395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:01.964395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:01.964395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:01.964395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:01.964395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:01.964395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:01.964395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:01.964395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:01.964395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:01.964395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:01.964395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:01.964395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:01.964395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:01.964877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:01.964877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:01.964877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.964877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:01.965041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:01.965362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:01.965824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:01.965824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:01.965824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:01.965824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:02.000580 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:02.001018 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:02.001186 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:02.001387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:02.002481 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:02.002763 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:02.003038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:02.003622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:02.003622 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:02.003622 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:02.003622 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:02.003622 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:02.003622 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:02.003622 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:02.003622 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:02.004731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:02.004731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:02.004731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:02.004731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:02.004731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:02.004731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:02.004731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:02.004731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:02.004731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:02.004731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:02.004731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:02.004731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:02.005279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:02.005279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:02.005279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:02.005279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:02.005279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:02.005279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:02.005279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:02.005279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:02.005279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:02.005279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:02.005279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:02.005279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:02.005279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:02.005779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:02.005779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:02.005779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:02.005779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:02.005779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:02.005779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:02.005779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:02.005779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:02.005779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:02.005779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:02.005779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:02.005779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:02.005779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:02.005779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:02.005779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:02.006341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:02.006936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:02.006936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:02.006936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:02.006936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:02.006936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:02.006936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:02.027750 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:02.028181 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:02.028361 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:02.029310 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:02.085306 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:02.085554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:02.086522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:02.086522 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:02.086522 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:02.086522 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:02.086522 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:02.086522 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:02.086522 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:02.086522 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:02.087218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.058 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:02.087218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:02.087218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:02.087218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:02.087218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:02.087218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:02.087218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:02.087218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:02.087218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:02.087218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:02.087218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:02.087218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:02.087672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:02.087672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:02.087672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:02.087672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:02.087672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:02.087672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:02.087672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:02.087672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:02.087672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:02.087672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:02.087672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:02.087672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:02.087672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:02.088181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:02.088181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:02.088278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:02.088278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:02.088278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:02.088278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:02.088278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:02.088278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:02.088278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:02.088278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:02.088278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:02.088278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:02.088278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:02.088278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:02.088278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:02.088278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:02.088278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:02.088278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:02.088710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:02.089176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:02.089176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:02.089176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:06.887613 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:06.887613 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:06.887613 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:06.888293 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:06.888657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:06.888844 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:06.888991 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:06.889133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:06.889269 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:06.890081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:06.890081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:06.890081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:06.890081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:06.890081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:06.890081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.890897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:06.890897 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:06.890897 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:06.890897 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:06.890897 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:06.890897 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:06.890897 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.890897 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:06.892086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.892086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:06.892086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:06.892086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:06.892086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:06.892086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:06.892086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:06.892086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:06.892086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:06.892086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:06.892086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:06.892086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:06.892378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:06.892378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:06.892378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:06.892378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:06.892378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:06.892378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:06.892378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:06.892378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:06.892378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:06.892378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:06.892378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:06.892378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:06.892378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:06.892673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.892673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:06.893298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.893298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:06.893298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:06.893298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:06.893298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:06.893298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:06.893298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:06.893298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:06.893298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:06.893298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:06.893298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:06.893298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:06.893298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:06.893298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:06.893298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:06.893298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:06.893641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:06.893980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:06.893980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.893980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:06.947520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:06.948725 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:06.948915 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:06.949068 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:40:06.950232 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:06.950529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.951290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:06.951290 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:06.951290 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:06.951290 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:06.951290 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:06.951290 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:06.951290 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.951290 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:06.952639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.952639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:06.952639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:06.952639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:06.952639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:06.952639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:06.952639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:06.952639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:06.952639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:06.952639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:06.952639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:06.952639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:06.952639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:06.952639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:06.952639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:06.953153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:06.953153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:06.953153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:06.953153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:06.953153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:06.953153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:06.953153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:06.953153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.953153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:06.953373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.953373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:06.953373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:06.953373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:06.953373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:06.953373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:06.953373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:06.953373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:40:06.953373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:06.953373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:06.953373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:06.953373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:06.953823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:06.953823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:06.953823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:06.953823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:06.953823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:06.953823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:06.953823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:06.953823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:06.953823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:06.953823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:06.953823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:06.953823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:06.953823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:06.953823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:06.954236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:06.954236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:06.954236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.954236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:06.960507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.960507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:06.960507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:06.960507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:06.960507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:06.960507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:06.960507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:06.960507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:06.960507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:06.960507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:06.960507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:06.960507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:06.960507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:06.960507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:06.960507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:06.960507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:06.961129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:06.961515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:06.961515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:06.961515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:06.961515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:06.961515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:06.961515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.961515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:06.961515 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:06.961515 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:06.961738 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:06.961944 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:06.966746 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:06.967056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.967654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:06.967654 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:06.967654 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:06.967654 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:06.967654 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:06.967654 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:06.967654 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.967654 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:06.968401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.968401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:06.968401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:06.968401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:06.968401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:06.968401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:06.968401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:06.968401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:06.968401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:06.968401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:06.968401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:06.968401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:06.968401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:06.968401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:06.968401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:06.968748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:06.968748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:06.968748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:06.968748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:06.968748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:06.968748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:06.968748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:06.968748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.968748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:06.969328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.969328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:06.969328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:06.969328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:06.969328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:06.969328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:06.969328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:06.969328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:06.969328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:06.969328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:06.969328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:06.969711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:06.969711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:06.969711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:06.969711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:06.969711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:06.969711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:06.969711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:06.969711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:06.969711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:06.969711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:06.969711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:06.969711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:06.969711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:06.969711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:06.970026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:06.970026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:06.970026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.970026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:06.970373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:06.970748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:06.971076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:06.971076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:06.971076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:06.971076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:07.002300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:07.002300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:07.002300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:07.002300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:07.002739 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:07.002739 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:07.003081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:07.003696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:07.003696 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:07.003696 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:07.003696 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:07.003696 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:07.003696 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:07.003696 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:07.003696 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:07.004629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:07.004629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:07.004629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:07.004629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:07.004629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:07.004629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:07.004629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:07.004629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:07.004629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:07.004629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:07.004629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:07.004629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:07.005036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:07.005036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:07.005036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:07.005036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:07.005036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:07.005036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:07.005036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:07.005036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:07.005036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:07.005036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:07.005036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:07.005036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:07.005036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:07.005399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:07.005399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:07.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:07.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:07.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:07.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:07.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:07.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:07.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:07.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:07.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:07.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:07.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:07.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:07.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:07.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:07.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:07.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:07.005954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:07.007710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:07.007710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:07.007710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:07.030389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:07.035639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:07.035639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:07.035639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:07.095559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:07.096889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:07.097697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:07.097697 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:07.097697 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:07.097697 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:07.097697 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:07.097697 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:07.097697 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:07.097697 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:07.098836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.067 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:07.098836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:07.098836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:07.098836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:07.098836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:07.098836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:07.098836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:07.098836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:07.098836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:07.098836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:07.098836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:07.098836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:07.099629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:07.099629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:07.099629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:07.099629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:07.099629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:07.099629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:07.099629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:07.099629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:07.099629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:07.099629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:07.099629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:07.099629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:07.099629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:07.100128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:07.100128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:07.100128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:07.100128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:07.100128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:07.100128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:07.100128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:07.100128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:07.100128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:07.100128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:07.100128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:07.100128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:07.100128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:07.100128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:07.100128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:07.100523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:07.101044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:07.101044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:07.101044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:07.101044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:07.101044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:07.101044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:11.887876 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:11.888357 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:11.888421 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:11.889594 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:11.889801 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:11.889936 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:11.890051 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:11.890157 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:11.890261 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:11.890364 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:11.890490 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:11.890624 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:11.890731 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:11.890834 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:11.890939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.891465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:11.891465 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:11.891465 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:11.891465 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:11.891465 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:11.891465 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:11.891465 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.891465 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:11.892225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.892225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:11.892225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:11.892225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:11.892225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:11.892225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:11.892225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:11.892225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:11.892225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:11.892225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:11.892225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:11.892225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:11.892621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:11.892621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:11.892621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:11.892621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:11.892621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:11.892621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:11.892621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:11.892621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:11.892621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:11.892621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:11.892621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:11.892621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:11.892621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:11.893013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.893013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:11.893013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.893013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:11.893013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:11.893013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:11.893013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:11.893013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:11.893013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:11.893013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:11.893013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:11.893013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:11.893013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:11.893013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:11.893013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:11.893439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:11.893995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:11.893995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:11.893995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:11.893995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:11.893995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.893995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:11.948718 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:11.948997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:11.949135 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:11.949281 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:40:11.949874 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:11.950095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.950714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:11.950714 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:11.950714 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:11.950714 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:11.950714 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:11.950714 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:11.950714 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.950714 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:11.953557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.953557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:11.953557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:11.953557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:11.953557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:11.953557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:11.953557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:11.953557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:11.953557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:11.953557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:11.953557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:11.953557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:11.953557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:11.953557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:11.953557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:11.953994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:11.953994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:11.953994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:11.953994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:11.953994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:11.953994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:11.953994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:11.953994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.953994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:11.954621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.954621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:11.954621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:11.954621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:11.954621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:11.954621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:11.954621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:11.954621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:40:11.954621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:11.954621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:11.954621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:11.954621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:11.955001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:11.955001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:11.955001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:11.955001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:11.955001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:11.955001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:11.955001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:11.955001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:11.955001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:11.955001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:11.955001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:11.955001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:11.955001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:11.955001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:11.955392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:11.955392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:11.955392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.955392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:11.955835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.955835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:11.955835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:11.955835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:11.955835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:11.955835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:11.955835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:11.955835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:11.955835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:11.955835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:11.955835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:11.955835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:11.955835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:11.955835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:11.955835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:11.955835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:11.956290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:11.956848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:11.956848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:11.956848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:11.956848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:11.956848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:11.956848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.956848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:11.961404 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:11.961775 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:11.962247 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:11.962247 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:11.963081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:11.964344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.964344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:11.964344 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:11.964344 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:11.964344 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:11.964344 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:11.964344 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:11.964344 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.964344 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:11.965082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.965082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:11.965082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:11.965082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:11.965082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:11.965082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:11.965082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:11.965082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:11.965082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:11.965082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:11.965082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:11.965082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:11.965082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:11.965082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:11.965082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:11.965517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:11.965517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:11.965517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:11.965517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:11.965517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:11.965517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:11.965517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:11.965517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.965517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:11.966109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.966109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:11.966109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:11.966109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:11.966109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:11.966109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:11.966109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:11.966109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:11.966109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:11.966109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:11.966109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:11.966787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:11.966787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:11.966787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:11.966787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:11.966787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:11.966787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:11.966787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:11.966787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:11.966787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:11.966787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:11.966787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:11.966787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:11.966787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:11.966787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:11.967222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:11.967222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:11.967222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.967222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:11.967379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:11.969447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:11.969810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:11.969810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:11.969810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:11.969810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:12.003796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:12.004048 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:12.004186 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:12.004357 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:12.004922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:12.005252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:12.005686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:12.006257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:12.006257 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:12.006257 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:12.006257 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:12.006257 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:12.006257 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:12.006257 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:12.006257 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:12.007087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:12.007087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:12.007087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:12.007087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:12.007087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:12.007087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:12.007087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:12.007087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:12.007087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:12.007087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:12.007087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:12.007087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:12.007464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:12.007464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:12.007464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:12.007464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:12.007464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:12.007464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:12.007464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:12.007464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:12.007464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:12.007464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:12.007464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:12.007464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:12.007464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:12.007832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:12.007832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:12.008161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:12.008161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:12.008161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:12.008161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:12.008161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:12.008161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:12.008161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:12.008161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:12.008161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:12.008161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:12.008161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:12.008161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:12.008161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:12.008161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:12.008161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:12.008161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:12.008513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:12.008940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:12.008940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:12.008940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:12.030885 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:12.031137 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:12.031266 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:12.031948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:12.076517 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:12.077318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:12.077511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:12.077511 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:12.077511 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:12.077511 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:12.077511 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:12.077511 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:12.077511 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:12.077511 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:12.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:12.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:12.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:12.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:12.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:12.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:12.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:12.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:12.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:12.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:12.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:12.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:12.079341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:12.079341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:12.079341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:12.079341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:12.079341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:12.079341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:12.079341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:12.079341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:12.079341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:12.079341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:12.079341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:12.079341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:12.079341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:12.079759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:12.079759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:12.081688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:12.081688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:12.081688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:12.081688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:12.081688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:12.081688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:12.081688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:12.081688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:12.081688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:12.081688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:12.081688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:12.081688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:12.081688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:12.081688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:12.081688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:12.081688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:12.082169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:12.082685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:12.082685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:12.082685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:16.888346 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:16.890286 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:16.890286 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:16.890372 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:16.890894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:16.890894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:16.890894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:16.891215 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:16.891387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:16.891608 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:16.891745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:16.891861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:16.891979 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:16.892081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:16.892178 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:16.892270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.894289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:16.894289 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:16.894289 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:16.894289 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:16.894289 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:16.894289 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:16.894289 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.894289 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:16.894289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.894289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:16.894289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:16.894289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:16.894289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:16.894289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:16.894289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:16.895368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:16.895368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:16.895368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:16.895368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:16.895368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:16.895368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:16.895368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:16.895368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:16.895368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:16.895368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:16.895368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:16.895368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:16.895368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:16.895368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:16.895741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:16.895741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:16.895741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:16.895741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:16.895741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.895741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:16.895741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.895741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:16.895741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:16.895741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:16.895741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:16.895741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:16.895741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:16.895741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:16.896026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:16.896510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:16.896510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:16.896510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:16.896510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:16.896510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:16.896510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:16.896510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:16.896510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:16.896510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.896510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:16.949186 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:16.950322 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:16.950322 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:16.950322 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:40:16.950783 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:16.950883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:16.954052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:16.954632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:16.954632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:16.954632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:16.954632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:16.954632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:16.954632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:16.954632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:16.954632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:16.954632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:16.954632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:16.954632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:16.954632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:16.954632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.954632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:16.954632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.954942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:16.954942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:16.954942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:16.954942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:16.954942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:16.954942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:16.954942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:40:16.954942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:16.954942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:16.954942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:16.954942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:16.954942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:16.954942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:16.955153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:16.955153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:16.955153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:16.955153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:16.955153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:16.955153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:16.955153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:16.955153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:16.955153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:16.955153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:16.955153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:16.955153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:16.955153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:16.955153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:16.955416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.955416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:16.955416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.955416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:16.955416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:16.955416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:16.955416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:16.955416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:16.955416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:16.955416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:16.955416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:16.955416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:16.955416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:16.955416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:16.955416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:16.955705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:16.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:16.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:16.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:16.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:16.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:16.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:16.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:16.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.956039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:16.961863 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:16.962621 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:16.962886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:16.963241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:16.963761 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:16.963997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:16.966833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:16.967260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:16.967260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:16.967260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:16.967260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:16.967260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:16.967260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:16.967260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:16.967260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:16.967260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:16.967260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:16.967260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:16.967260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:16.967260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.967260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:16.967260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.967570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:16.967570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:16.967570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:16.967570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:16.967570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:16.967570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:16.967570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:16.967570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:16.967570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:16.967570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:16.967570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:16.967570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:16.967570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:16.967817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:16.967817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:16.967817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:16.967817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:16.967817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:16.967817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:16.967817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:16.967817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:16.967817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:16.967817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:16.967817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:16.967817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:16.967817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:16.968131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.968131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:16.969522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:16.971763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:16.972967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:16.972967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:16.972967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:16.972967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:17.004828 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:17.005339 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:17.008320 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:17.009019 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:17.010225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:17.010458 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:17.010820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:17.011927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:17.011927 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:17.011927 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:17.011927 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:17.011927 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:17.011927 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:17.011927 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:17.011927 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:17.014223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:17.014223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:17.014223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:17.014223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:17.014223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:17.014223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:17.014223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:17.014223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:17.014223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:17.014223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:17.014223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:17.014223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:17.014560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:17.014560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:17.014560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:17.014560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:17.014560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:17.014560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:17.014560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:17.014560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:17.014560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:17.014560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:17.014560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:17.014560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:17.014560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:17.014873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:17.014873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:17.014873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:17.014873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:17.014873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:17.014873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:17.014873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:17.014873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:17.014873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:17.014873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:17.014873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:17.014873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:17.014873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:17.014873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:17.014873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:17.015258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:17.015774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:17.015774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:17.015774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:17.015774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:17.015774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:17.015774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:17.031390 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:17.033510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:17.033510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:17.033510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:17.124181 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:17.124427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:17.125482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:17.125482 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:17.125482 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:17.125482 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:17.125482 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:17.125482 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:17.125482 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:17.125482 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:17.126447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.093 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:17.126447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:17.126447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:17.126447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:17.126447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:17.126447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:17.126447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:17.126447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:17.126447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:17.126447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:17.126447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:17.126447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:17.126797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:17.126797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:17.126797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:17.126797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:17.126797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:17.126797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:17.126797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:17.126797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:17.126797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:17.126797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:17.126797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:17.126797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:17.126797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:17.127071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:17.127071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:17.127638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:17.127638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:17.127638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:17.127638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:17.127638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:17.127638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:17.127638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:17.127638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:17.127638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:17.127638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:17.127638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:17.127638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:17.127638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:17.127638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:17.127638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:17.127638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:17.127988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:17.128935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:17.128935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:17.128935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:21.889743 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:21.892139 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:21.892139 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:21.893072 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:21.893387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:21.893686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:21.893926 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:21.894174 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:21.894436 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:21.894686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:21.895262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:21.895509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:21.895747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:21.896009 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:21.896219 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:21.896576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.897287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:21.897287 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:21.897287 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:21.897287 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:21.897287 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:21.897287 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:21.897287 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.897287 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:21.898639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.898639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:21.898639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:21.898639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:21.898639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:21.898639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:21.898639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:21.898639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:21.898639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:21.898639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:21.898639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:21.898639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:21.899033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:21.899033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:21.899033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:21.899033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:21.899033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:21.899033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:21.899033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:21.899033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:21.899033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:21.899033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:21.899033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:21.899033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:21.899033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:21.899475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.899475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:21.903038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.903038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:21.903038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:21.903038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:21.903038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:21.903038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:21.903038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:21.903038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:21.903038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:21.903038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:21.903038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:21.903038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:21.903038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:21.903038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:21.903038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:21.903038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:21.903746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:21.904237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:21.904237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.904237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:21.951082 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:21.954569 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:21.954569 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:21.954569 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:40:21.958575 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:21.963071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.963071 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:21.965036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:21.965036 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:21.965036 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:21.965036 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:21.965036 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:21.965036 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:21.965036 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.965036 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:21.969575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.969575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:21.969575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:21.969575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:21.969575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:21.969575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:21.969575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:21.969575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:21.969575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:21.969575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:21.969575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:21.969575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:21.969575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:21.969575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:21.969575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:21.970348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:21.970348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:21.970348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:21.970348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:21.970348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:21.970348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:21.970348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:21.970348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.970348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:21.970348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.015 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.970348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:21.970348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:21.970348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:21.970348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:21.970857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:21.970857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:21.970857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:40:21.970857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:21.970857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:21.970857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:21.970857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:21.970857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:21.970857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:21.970857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:21.970857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:21.970857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:21.970857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:21.971226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:21.971226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:21.971226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:21.971226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:21.971226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:21.971226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:21.971226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:21.971226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:21.971226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:21.971226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:21.971226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.971226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:21.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:21.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:21.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:21.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:21.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:21.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:21.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:21.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:21.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:21.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:21.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:21.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:21.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:21.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:21.973107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:21.973785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:21.974595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:21.974595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:21.974595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:21.974595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:21.974595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:21.974595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.974595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:21.974595 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:21.974595 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:21.974595 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:21.974891 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:21.975261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:21.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:21.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:21.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:21.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:21.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:21.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:21.977665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.977665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:21.977665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:21.977665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:21.977665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:21.977665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:21.977665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:21.977665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:21.977665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:21.977665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:21.977665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:21.977665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:21.977665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:21.977665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:21.977665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:21.978140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:21.978140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:21.978140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:21.978140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:21.978140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:21.978140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:21.978140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:21.978140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.978140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:21.979634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.979634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:21.979634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:21.979634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:21.979634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:21.979634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:21.979634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:21.979634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:21.979634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:21.979634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:21.979634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:21.979929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:21.979929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:21.979929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:21.979929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:21.979929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:21.979929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:21.979929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:21.979929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:21.979929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:21.979929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:21.979929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:21.979929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:21.979929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:21.979929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:21.980348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:21.980348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:21.980348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.980348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:21.980532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:21.981058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:21.981617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:21.981617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:21.981617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:21.981617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:22.005758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:22.006095 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:22.006274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:22.006504 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:22.006940 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:22.007145 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:22.007894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:22.009167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:22.009167 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:22.009167 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:22.009167 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:22.009167 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:22.009167 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:22.009167 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:22.009167 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:22.010455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:22.010455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:22.010455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:22.010455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:22.010455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:22.010455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:22.010455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:22.010455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:22.010455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:22.010455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:22.010455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:22.010455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:22.010852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:22.010852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:22.010852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:22.010852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:22.010852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:22.010852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:22.010852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:22.010852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:22.010852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:22.010852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:22.010852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:22.010852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:22.010852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:22.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:22.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:22.011561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:22.011561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:22.011561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:22.011561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:22.011561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:22.011561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:22.011561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:22.011561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:22.011561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:22.011561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:22.011561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:22.011561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:22.011561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:22.011561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:22.011561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:22.011561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:22.012148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:22.013090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:22.013090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:22.013090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:22.031958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:22.032327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:22.032526 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:22.033505 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:22.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=242 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:40:22.087916 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:22.088192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:22.088907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:22.088907 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:22.088907 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:22.088907 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:22.088907 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:22.088907 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:22.088907 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:22.088907 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:22.090136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:22.090136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:22.090136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:22.090136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:22.090136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:22.090136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:22.090136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:22.090136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:22.090136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:22.090136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:22.090136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:22.090136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:22.090827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:22.090827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:22.090827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:22.090827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:22.090827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:22.090827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:22.090827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:22.090827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:22.090827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:22.090827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:22.090827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:22.090827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:22.090827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:22.091052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:22.091052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:22.091335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:22.091335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:22.091335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:22.091335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:22.091335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:22.091335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:22.091335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:22.091335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:22.091335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:22.091335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:22.091335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:22.091335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:22.091335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:22.091335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:22.091335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:22.091335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:22.091682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:22.092190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:22.092190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:22.092190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:26.890612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:26.892773 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:26.892773 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:26.894996 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:26.895160 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:26.895299 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:26.895437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:26.895576 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:26.895718 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:26.895914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:26.896038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:26.896160 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:26.896321 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:26.896456 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:26.896595 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:26.896715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.897337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:26.897337 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:26.897337 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:26.897337 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:26.897337 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:26.897337 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:26.897337 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.897337 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:26.898346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.898346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:26.898346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:26.898346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:26.898346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:26.898346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:26.898346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:26.898346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:26.898346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:26.898346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:26.898346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:26.898346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:26.898699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:26.898699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:26.898699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:26.898699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:26.898699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:26.898699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:26.898699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:26.898699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:26.898699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:26.898699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:26.898699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:26.898699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:26.898699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:26.898941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.898941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:26.900274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.900274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:26.900274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:26.900274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:26.900274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:26.900274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:26.900274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:26.900274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:26.900274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:26.900274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:26.900274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:26.900274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:26.900274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:26.900274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:26.900274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:26.900274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:26.900737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:26.901104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:26.901104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.901104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:26.952528 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:26.952990 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:26.953148 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:26.953277 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:40:26.953734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:26.953914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.954336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:26.954336 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:26.954336 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:26.954336 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:26.954336 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:26.954336 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:26.954336 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.954336 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:26.954957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.954957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:26.954957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:26.954957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:26.954957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:26.954957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:26.954957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:26.954957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:26.954957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:26.954957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:26.954957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:26.954957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:26.954957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:26.954957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:26.954957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:26.955281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:26.955281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:26.955281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:26.955281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:26.955281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:26.955281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:26.955281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:26.955281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.955281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:26.955610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.955610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:26.955610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:26.955610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:26.955610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:26.955610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:26.955610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:26.955610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:40:26.955610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:26.955610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:26.955610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:26.955610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:26.955836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:26.955836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:26.955836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:26.955836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:26.955836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:26.955836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:26.955836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:26.955836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:26.955836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:26.955836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:26.955836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:26.955836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:26.955836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:26.955836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:26.956071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:26.956071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:26.956071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.956071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:26.956317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.956317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:26.956317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:26.956317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:26.956317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:26.956317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:26.956317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:26.956317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:26.956317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:26.956317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:26.956317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:26.956317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:26.956317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:26.956317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:26.956317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:26.956317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:26.956642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:26.956944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:26.956944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:26.956944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:26.956944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:26.956944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:26.956944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.956944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:26.973190 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:26.973758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:26.973876 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:26.974564 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:26.974564 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:26.974684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.975130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:26.975130 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:26.975130 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:26.975130 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:26.975130 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:26.975130 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:26.975130 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.975130 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:26.975890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.975890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:26.975890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:26.975890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:26.975890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:26.975890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:26.975890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:26.975890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:26.975890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:26.975890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:26.975890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:26.975890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:26.975890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:26.975890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:26.975890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:26.976384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:26.976384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:26.976384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:26.976384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:26.976384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:26.976384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:26.976384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:26.976384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.976384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:26.976817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.976817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:26.976817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:26.976817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:26.976817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:26.976817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:26.976817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:26.976817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:26.976817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:26.976817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:26.976817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:26.977159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:26.977159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:26.977159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:26.977159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:26.977159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:26.977159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:26.977159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:26.977159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:26.977159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:26.977159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:26.977159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:26.977159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:26.977159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:26.977159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:26.977614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:26.977614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:26.977614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.977614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:26.977889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:26.978422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:26.978962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:26.978962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:26.978962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:26.978962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:27.006336 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:27.006794 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:27.006938 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:27.007166 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:27.007486 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:27.007743 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:27.008101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:27.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:27.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:27.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:27.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:27.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:27.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:27.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:27.008653 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:27.009728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:27.009728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:27.009728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:27.009728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:27.009728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:27.009728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:27.009728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:27.009728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:27.009728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:27.009728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:27.009728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:27.009728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:27.009952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:27.009952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:27.009952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:27.009952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:27.009952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:27.009952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:27.009952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:27.009952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:27.009952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:27.009952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:27.009952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:27.009952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:27.009952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:27.010180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:27.010180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:27.010817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:27.010817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:27.010817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:27.010817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:27.010817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:27.010817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:27.010817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:27.010817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:27.010817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:27.010817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:27.010817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:27.010817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:27.010817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:27.010817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:27.010817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:27.010817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:27.011095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:27.011392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:27.011392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:27.011392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:27.032325 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:27.032779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:27.032930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:27.033838 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:27.077203 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:27.077487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:27.078123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:27.078123 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:27.078123 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:27.078123 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:27.078123 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:27.078123 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:27.078123 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:27.078123 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:27.079008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:27.079008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:27.079008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:27.079008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:27.079008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:27.079008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:27.079008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:27.079008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:27.079008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:27.079008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:27.079008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:27.079008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:27.079291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:27.079291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:27.079291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:27.079291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:27.079291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:27.079291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:27.079291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:27.079291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:27.079291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:27.079291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:27.079291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:27.079291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:27.079291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:27.079569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:27.079569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:27.080151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:27.080151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:27.080151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:27.080151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:27.080151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:27.080151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:27.080151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:27.080151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:27.080151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:27.080151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:27.080151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:27.080151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:27.080151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:27.080151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:27.080151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:27.080151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:27.080500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:27.080904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:27.080904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:27.080904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:31.892022 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:31.892500 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:31.892500 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:31.893563 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:31.893728 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:31.893851 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:31.893974 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:31.894079 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:31.894177 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:31.894309 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:31.894432 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:31.894532 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:31.894677 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:31.894784 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:31.894894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:31.895005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.895504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:31.895504 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:31.895504 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:31.895504 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:31.895504 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:31.895504 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:31.895504 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.895504 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:31.896163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.896163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:31.896163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:31.896163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:31.896163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:31.896163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:31.896163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:31.896163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:31.896163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:31.896163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:31.896163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:31.896163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:31.896389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:31.896389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:31.896389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:31.896389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:31.896389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:31.896389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:31.896389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:31.896389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:31.896389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:31.896389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:31.896389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:31.896389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:31.896389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:31.896657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.896657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:31.896918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.896918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:31.896918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:31.896918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:31.896918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:31.896918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:31.896918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:31.896918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:31.896918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:31.896918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:31.896918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:31.896918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:31.896918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:31.896918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:31.896918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:31.896918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:31.897263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:31.897613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:31.897613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.897613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:31.953060 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:31.953357 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:31.953522 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:31.953651 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:40:31.954045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:31.954248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.954863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:31.954863 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:31.954863 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:31.954863 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:31.954863 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:31.954863 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:31.954863 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.954863 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:31.955383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.955383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:31.955383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:31.955383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:31.955383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:31.955383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:31.955383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:31.955383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:31.955383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:31.955383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:31.955383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:31.955383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:31.955383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:31.955383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:31.955383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:31.955834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:31.955834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:31.955834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:31.955834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:31.955834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:31.955834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:31.955834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:31.955834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.955834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:31.956124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.956124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:31.956124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:31.956124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:31.956124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:31.956124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:31.956124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:31.956124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:40:31.956124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:31.956124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:31.956124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:31.956124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:31.956345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:31.956345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:31.956345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:31.956345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:31.956345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:31.956345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:31.956345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:31.956345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:31.956345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:31.956345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:31.956345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:31.956345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:31.956345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:31.956345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:31.956663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:31.956663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:31.956663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.956663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:31.956853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.956853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:31.956853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:31.956853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:31.956853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:31.956853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:31.956853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:31.956853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:31.956853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:31.956853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:31.956853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:31.956853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:31.956853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:31.956853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:31.956853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:31.956853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:31.957139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:31.957577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:31.957577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:31.957577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:31.957577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:31.957577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:31.957577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.957577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:31.973776 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:31.974040 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:31.974141 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:31.974249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:31.974607 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:31.974797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.975203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:31.975203 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:31.975203 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:31.975203 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:31.975203 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:31.975203 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:31.975203 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.975203 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:31.975804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.975804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:31.975804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:31.975804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:31.975804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:31.975804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:31.975804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:31.975804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:31.975804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:31.975804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:31.975804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:31.975804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:31.975804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:31.975804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:31.975804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:31.976073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:31.976073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:31.976073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:31.976073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:31.976073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:31.976073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:31.976073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:31.976073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.976073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:31.976440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.976440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:31.976440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:31.976440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:31.976440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:31.976440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:31.976440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:31.976440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:31.976440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:31.976440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:31.976440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:31.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:31.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:31.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:31.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:31.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:31.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:31.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:31.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:31.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:31.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:31.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:31.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:31.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:31.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:31.976906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:31.976906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:31.976906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.976906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:31.977232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:31.977604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:31.977969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:31.977969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:31.977969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:31.977969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:32.006789 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:32.007095 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:32.007242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:32.007425 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:32.007735 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:32.007919 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:32.008173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:32.008627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:32.008627 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:32.008627 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:32.008627 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:32.008627 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:32.008627 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:32.008627 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:32.008627 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:32.009322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:32.009322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:32.009322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:32.009322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:32.009322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:32.009322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:32.009322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:32.009322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:32.009322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:32.009322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:32.009322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:32.009322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:32.009595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:32.009595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:32.009595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:32.009595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:32.009595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:32.009595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:32.009595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:32.009595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:32.009595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:32.009595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:32.009595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:32.009595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:32.009595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:32.009893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:32.009893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:32.010066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:32.010066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:32.010066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:32.010066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:32.010066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:32.010066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:32.010066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:32.010066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:32.010066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:32.010066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:32.010066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:32.010066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:32.010066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:32.010066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:32.010066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:32.010066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:32.010345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:32.010783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:32.010783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:32.010783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:32.032797 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:32.033096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:32.033219 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:32.033978 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:32.076157 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:32.076447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:32.077036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:32.077036 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:32.077036 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:32.077036 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:32.077036 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:32.077036 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:32.077036 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:32.077036 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:32.077929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:32.077929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:32.077929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:32.077929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:32.077929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:32.077929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:32.077929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:32.077929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:32.077929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:32.077929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:32.077929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:32.077929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:32.078179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:32.078179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:32.078179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:32.078179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:32.078179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:32.078179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:32.078179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:32.078179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:32.078179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:32.078179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:32.078179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:32.078179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:32.078179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:32.078418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:32.078418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:32.078757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:32.078757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:32.078757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:32.078757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:32.078757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:32.078757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:32.078757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:32.078757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:32.078757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:32.078757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:32.078757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:32.078757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:32.078757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:32.078757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:32.078757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:32.078757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:32.079136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:32.079526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:32.079526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:32.079526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:36.892494 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:36.892962 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:36.892962 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:36.893783 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:36.893965 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:36.894348 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:36.894348 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:36.894348 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:36.894490 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:36.894594 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:36.894702 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:36.894842 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:36.894905 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:36.895008 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:36.895105 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:36.895205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.897740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:36.897740 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:36.897740 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:36.897740 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:36.897740 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:36.897740 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:36.897740 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.897740 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:36.898444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.898444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:36.898444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:36.898444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:36.898444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:36.898444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:36.898444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:36.898444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:36.898444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:36.898444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:36.898444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:36.898444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:36.900012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:36.900012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:36.900012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:36.900012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:36.900012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:36.900012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:36.900012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:36.900012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:36.900012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:36.900012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:36.900012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:36.900012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:36.900012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:36.900257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.900257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:36.900257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.900257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:36.900257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:36.900257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:36.900257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:36.900257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:36.900257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:36.900257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:36.900257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:36.900257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:36.900257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:36.900257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:36.900257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:36.900584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:36.900904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:36.900904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:36.900904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:36.900904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:36.900904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.900904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:36.953614 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:36.955218 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:36.955218 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:36.955218 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:40:36.955218 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:36.955218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.956900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:36.956900 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:36.956900 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:36.956900 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:36.956900 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:36.956900 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:36.956900 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.956900 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:36.958856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.958856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:36.958856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:36.958856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:36.958856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:36.958856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:36.958856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:36.958856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:36.958856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:36.958856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:36.958856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:36.958856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:36.958856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:36.958856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:36.958856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:36.959435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:36.959435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:36.959435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:36.959435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:36.959435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:36.959435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:36.959435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:36.959435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.959435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:36.959435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.959435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:36.959435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:36.959435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:36.959435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:36.959883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:36.959883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:36.959883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:40:36.959883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:36.959883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:36.959883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:36.959883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:36.959883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:36.959883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:36.959883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:36.959883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:36.959883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:36.959883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:36.960233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:36.960233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:36.960233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:36.960233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:36.960233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:36.960233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:36.960233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:36.960233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:36.960233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:36.960233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:36.960233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.960233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:36.960575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.960575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:36.960575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:36.960575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:36.960575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:36.960575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:36.960575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:36.960575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:36.960575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:36.960575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:36.960575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:36.960575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:36.960575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:36.960575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:36.960575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:36.960575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:36.961121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:36.961670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:36.961670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:36.961670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:36.961670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:36.961670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:36.961670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.961670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:36.974204 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:36.974668 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:36.974800 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:36.974913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:36.975523 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:36.975908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.976437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:36.976437 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:36.976437 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:36.976437 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:36.976437 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:36.976437 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:36.976437 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.976437 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:36.977186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.977186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:36.977186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:36.977186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:36.977186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:36.977186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:36.977186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:36.977186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:36.977186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:36.977186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:36.977186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:36.977186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:36.977186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:36.977186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:36.977186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:36.977477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:36.977477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:36.977477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:36.977477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:36.977477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:36.977477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:36.977477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:36.977477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.977477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:36.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:36.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:36.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:36.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:36.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:36.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:36.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:36.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:36.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:36.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:36.978416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:36.978416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:36.978416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:36.978416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:36.978416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:36.978416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:36.978416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:36.978416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:36.978416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:36.978416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:36.978416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:36.978416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:36.978416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:36.978416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:36.978663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:36.978663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:36.978663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.978663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:36.979149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:36.979454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:36.979824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:36.979824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:36.979824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:36.979824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:37.007235 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:37.007874 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:37.007973 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:37.008252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:37.008690 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:37.008951 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:37.009354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:37.009976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:37.009976 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:37.009976 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:37.009976 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:37.009976 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:37.009976 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:37.009976 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:37.009976 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:37.010830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:37.010830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:37.010830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:37.010830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:37.010830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:37.010830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:37.010830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:37.010830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:37.010830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:37.010830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:37.010830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:37.010830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:37.011195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:37.011195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:37.011195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:37.011195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:37.011195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:37.011195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:37.011195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:37.011195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:37.011195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:37.011195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:37.011195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:37.011195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:37.011195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:37.011473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:37.011473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:37.011954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:37.011954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:37.011954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:37.011954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:37.011954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:37.011954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:37.011954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:37.011954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:37.011954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:37.011954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:37.011954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:37.011954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:37.011954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:37.011954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:37.011954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:37.011954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:37.012401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:37.012907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:37.012907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:37.012907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:37.033321 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:37.033791 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:37.033946 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:37.034645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:37.074072 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:37.074286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:37.075059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:37.075059 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:37.075059 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:37.075059 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:37.075059 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:37.075059 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:37.075059 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:37.075059 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:37.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:37.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:37.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:37.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:37.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:37.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:37.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:37.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:37.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:37.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:37.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:37.075693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:37.076136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:37.076136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:37.076136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:37.076136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:37.076136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:37.076136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:37.076136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:37.076136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:37.076136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:37.076136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:37.076136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:37.076136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:37.076136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:37.076430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:37.076430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:37.076430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:37.076430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:37.076430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:37.076430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:37.076430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:37.076430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:37.076430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:37.076430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:37.076430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:37.076430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:37.076430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:37.076430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:37.076430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:37.076763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:37.077181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:37.077181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:37.077181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:37.077181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:37.077181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:37.077181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:41.892867 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:41.893488 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:41.893488 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:41.894728 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:41.894995 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:41.895166 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:41.895335 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:41.895510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:41.895687 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:41.895826 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:41.895960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:41.896091 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:41.896225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:41.896365 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:41.896509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:41.896685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.897331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:41.897331 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:41.897331 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:41.897331 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:41.897331 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:41.897331 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:41.897331 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.897331 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:41.898105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.898105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:41.898105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:41.898105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:41.898105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:41.898105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:41.898105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:41.898105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:41.898105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:41.898105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:41.898105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:41.898105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:41.898568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:41.898568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:41.898568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:41.898568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:41.898568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:41.898568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:41.898568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:41.898568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:41.898568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:41.898568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:41.898568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:41.898568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:41.898568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:41.898916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.898916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:41.898982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.898982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:41.898982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:41.898982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:41.898982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:41.898982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:41.898982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:41.898982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:41.898982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:41.898982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:41.898982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:41.898982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:41.898982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:41.898982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:41.898982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:41.898982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:41.899381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:41.899903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:41.899903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.899903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:41.954080 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:41.954463 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:41.954598 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:41.954704 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:40:41.955290 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:41.955632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.956250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:41.956250 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:41.956250 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:41.956250 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:41.956250 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:41.956250 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:41.956250 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.956250 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:41.956921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.956921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:41.956921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:41.956921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:41.956921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:41.956921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:41.956921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:41.956921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:41.956921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:41.956921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:41.956921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:41.956921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:41.956921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:41.956921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:41.956921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:41.957316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:41.957316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:41.957316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:41.957316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:41.957316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:41.957316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:41.957316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:41.957316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.957316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:41.957594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.957594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:41.957594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:41.957594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:41.957594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:41.957594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:41.957594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:41.957594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:40:41.957594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:41.957594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:41.957594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:41.957594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:41.957912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:41.957912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:41.957912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:41.957912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:41.957912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:41.957912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:41.957912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:41.957912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:41.957912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:41.957912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:41.957912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:41.957912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:41.957912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:41.957912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:41.958269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:41.958269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:41.958269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.958269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:41.958379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.958379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:41.958379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:41.958379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:41.958379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:41.958379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:41.958379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:41.958379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:41.958379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:41.958379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:41.958379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:41.958379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:41.958379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:41.958379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:41.958379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:41.958379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:41.958879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:41.959323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:41.959323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:41.959323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:41.959323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:41.959323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:41.959323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.959323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:41.974698 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:41.975116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:41.975257 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:41.975392 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:41.975851 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:41.976117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.976708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:41.976708 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:41.976708 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:41.976708 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:41.976708 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:41.976708 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:41.976708 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.976708 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:41.978310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.978310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:41.978310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:41.978310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:41.978310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:41.978310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:41.978310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:41.978310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:41.978310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:41.978310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:41.978310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:41.978310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:41.978310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:41.978310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:41.978310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:41.978779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:41.978779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:41.978779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:41.978779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:41.978779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:41.978779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:41.978779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:41.978779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.978779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:41.978779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.978779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:41.978779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:41.978779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:41.978779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:41.979232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:41.979232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:41.979232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:41.979232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:41.979232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:41.979232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:41.979232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:41.979232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:41.979232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:41.979232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:41.979232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:41.979232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:41.979232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:41.979736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:41.979736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:41.979736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:41.979736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:41.979736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:41.979736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:41.979736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:41.979736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:41.979736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:41.979736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.979736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:41.979736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:41.980103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:41.980745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:42.007992 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:42.008385 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:42.008507 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:42.008710 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:42.009167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:42.009452 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:42.009839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:42.010408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:42.010408 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:42.010408 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:42.010408 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:42.010408 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:42.010408 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:42.010408 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:42.010408 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:42.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:42.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:42.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:42.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:42.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:42.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:42.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:42.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:42.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:42.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:42.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:42.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:42.011618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:42.011618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:42.011618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:42.011618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:42.011618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:42.011618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:42.011618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:42.011618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:42.011618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:42.011618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:42.011618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:42.011618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:42.011618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:42.012023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:42.012023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:42.012023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:42.012023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:42.012023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:42.012023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:42.012023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:42.012023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:42.012023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:42.012023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:42.012023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:42.012023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:42.012023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:42.012023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:42.012023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:42.012494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:42.013079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:42.013079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:42.013079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:42.013079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:42.013079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:42.013079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:42.033798 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:42.034194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:42.034309 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:42.035070 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:42.087117 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:42.087355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:42.089263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:42.089263 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:42.089263 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:42.089263 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:42.089263 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:42.089263 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:42.089263 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:42.089263 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:42.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:42.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:42.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:42.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:42.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:42.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:42.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:42.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:42.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:42.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:42.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:42.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:42.090779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:42.090779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:42.090779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:42.090779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:42.090779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:42.090779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:42.090779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:42.090779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:42.090779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:42.090779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:42.090779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:42.090779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:42.090779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:42.091170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:42.091170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:42.091356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:42.091356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:42.091356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:42.091356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:42.091356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:42.091356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:42.091356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:42.091356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:42.091356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:42.091356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:42.091356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:42.091356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:42.091356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:42.091356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:42.091356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:42.091356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:42.092058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:42.092799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:42.092799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:42.092799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:46.893467 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:46.894184 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:46.894184 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:46.895578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:46.895904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:46.896132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:46.896316 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:46.896505 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:46.896714 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:46.896860 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:46.897004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:46.897161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:46.897318 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:46.897489 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:46.897665 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:46.898471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.899275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:46.899275 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:46.899275 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:46.899275 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:46.899275 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:46.899275 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:46.899275 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.899275 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:46.900123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.900123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:46.900123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:46.900123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:46.900123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:46.900123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:46.900123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:46.900123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:46.900123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:46.900123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:46.900123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:46.900123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:46.900770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:46.900770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:46.900770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:46.900770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:46.900770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:46.900770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:46.900770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:46.900770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:46.900770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:46.900770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:46.900770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:46.900770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:46.900770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:46.901426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.901426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:46.901426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.901426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:46.901426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:46.901426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:46.901426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:46.901426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:46.901426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:46.901426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:46.901426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:46.901426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:46.901426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:46.901426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:46.901426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:46.902104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:46.903068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:46.903068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:46.903068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:46.903068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:46.903068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.903068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:46.954707 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:46.955137 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:46.955295 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:46.955468 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:40:46.955849 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:46.956033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.957313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:46.957313 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:46.957313 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:46.957313 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:46.957313 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:46.957313 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:46.957313 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.957313 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:46.957968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.957968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:46.957968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:46.957968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:46.957968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:46.957968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:46.957968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:46.957968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:46.957968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:46.957968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:46.957968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:46.957968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:46.957968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:46.957968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:46.957968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:46.958525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:46.958525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:46.958525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:46.958525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:46.958525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:46.958525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:46.958525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:46.958525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.958525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:46.958525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.958525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:46.958525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:46.958525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:46.958525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:46.958946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:46.958946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:46.958946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:40:46.958946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:46.958946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:46.958946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:46.958946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:46.958946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:46.958946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:46.958946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:46.958946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:46.958946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:46.958946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:46.959320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:46.959320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:46.959320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:46.959320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:46.959320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:46.959320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:46.959320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:46.959320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:46.959320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:46.959320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:46.959320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.959320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:46.959679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.959679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:46.959679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:46.959679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:46.959679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:46.959679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:46.959679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:46.959679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:46.959679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:46.959679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:46.959679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:46.959679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:46.959679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:46.959679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:46.959679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:46.959679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:46.960117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:46.960719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:46.960719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:46.960719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:46.960719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:46.960719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:46.960719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.960719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:46.975290 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:46.976254 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:46.976358 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:46.976440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:46.976880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:46.977090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.978745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:46.978745 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:46.978745 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:46.978745 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:46.978745 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:46.978745 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:46.978745 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.978745 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:46.979341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.979341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:46.979341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:46.979341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:46.979341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:46.979341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:46.979341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:46.979341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:46.979341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:46.979341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:46.979341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:46.979341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:46.979341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:46.979341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:46.979341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:46.979810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:46.979810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:46.979810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:46.979810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:46.979810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:46.979810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:46.979810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:46.979810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.979810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:46.980041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.980041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:46.980041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:46.980041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:46.980041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:46.980041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:46.980041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:46.980041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:46.980041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:46.980041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:46.980041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:46.980344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:46.980344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:46.980344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:46.980344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:46.980344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:46.980344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:46.980344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:46.980344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:46.980344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:46.980344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:46.980344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:46.980344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:46.980344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:46.980344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:46.980770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:46.980770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:46.980770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.980770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:46.980770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.980770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:46.980770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:46.980770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:46.980770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:46.980770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:46.980770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:46.980770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:46.980770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:46.980770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:46.981096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:46.981602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:46.981602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:46.981602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:46.981602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:46.981602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:46.981602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:46.981602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:46.981602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:46.981602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:46.981602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:46.981602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:47.008533 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:47.009673 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:47.009762 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:47.009922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:47.010340 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:47.023931 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:47.023931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:47.024183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:47.024183 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:47.024183 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:47.024183 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:47.024183 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:47.024183 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:47.024183 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:47.024183 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:47.027561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.014 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:47.027561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:47.027561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:47.027561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:47.027561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:47.027561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:47.027561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:47.027561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:47.027561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:47.027561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:47.027561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:47.027561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:47.028400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:47.028400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:47.028400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:47.028400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:47.028400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:47.028400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:47.028400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:47.028400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:47.028400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:47.028400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:47.028400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:47.028400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:47.028400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:47.028817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:47.028817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:47.028817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:47.028817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:47.028817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:47.028817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:47.028817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:47.028817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:47.028817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:47.028817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:47.028817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:47.028817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:47.028817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:47.028817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:47.028817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:47.029326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:47.029931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:47.029931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:47.029931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:47.029931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:47.029931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:47.029931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:47.035503 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:47.035503 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:47.035503 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:47.035703 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:47.075680 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:47.076065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:47.076643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:47.076643 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:47.076643 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:47.076643 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:47.076643 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:47.076643 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:47.076643 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:47.076643 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:47.077551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:47.077551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:47.077551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:47.077551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:47.077551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:47.077551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:47.077551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:47.077551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:47.077551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:47.077551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:47.077551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:47.077551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:47.077977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:47.077977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:47.077977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:47.077977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:47.077977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:47.077977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:47.077977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:47.077977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:47.077977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:47.077977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:47.077977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:47.077977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:47.077977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:47.078350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:47.078350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:47.079069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:47.079069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:47.079069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:47.079069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:47.079069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:47.079069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:47.079069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:47.079069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:47.079069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:47.079069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:47.079069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:47.079069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:47.079069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:47.079069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:47.079069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:47.079069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:47.079870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:47.080420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:47.080420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:47.080420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:51.893782 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:51.895951 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:51.895951 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:51.896269 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:51.896516 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:51.896745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:51.896939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:51.897125 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:51.897308 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:51.897504 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:51.897723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:51.897923 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:51.898109 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:51.898313 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:51.898514 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:51.898729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.899188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:51.899188 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:51.899188 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:51.899188 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:51.899188 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:51.899188 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:51.899188 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.899188 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:51.900156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.900156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:51.900156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:51.900156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:51.900156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:51.900156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:51.900156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:51.900156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:51.900156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:51.900156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:51.900156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:51.900156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:51.900614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:51.900614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:51.900614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:51.900614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:51.900614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:51.900614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:51.900614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:51.900614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:51.900614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:51.900614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:51.900614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:51.900614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:51.900614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:51.901890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.901890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:51.901890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.901890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:51.901890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:51.901890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:51.901890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:51.901890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:51.901890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:51.901890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:51.901890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:51.901890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:51.901890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:51.901890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:51.901890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:51.902395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:51.903037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:51.903037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:51.903037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:51.903037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:51.903037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.903037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:51.956822 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:51.958887 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:51.959120 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:51.959326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:40:51.959735 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:51.960029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.961706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:51.961706 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:51.961706 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:51.961706 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:51.961706 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:51.961706 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:51.961706 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.961706 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:51.962797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.962797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:51.962797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:51.962797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:51.962797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:51.962797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:51.962797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:51.962797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:51.962797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:51.962797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:51.962797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:51.962797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:51.962797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:51.962797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:51.962797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:51.963598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:51.963598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:51.963598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:51.963598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:51.963598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:51.963598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:51.963598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:51.963598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.963598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:51.964691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.964691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:51.964691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:51.964691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:51.964691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:51.964691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:51.964691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:51.964691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:40:51.964691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:51.964691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:51.964691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:51.964691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:51.965337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:51.965337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:51.965337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:51.965337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:51.965337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:51.965337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:51.965337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:51.965337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:51.965337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:51.965337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:51.965337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:51.965337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:51.965337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:51.965337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:51.969636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:51.969636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:51.969636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.969636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:51.971019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.971019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:51.971019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:51.971019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:51.971019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:51.971019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:51.971019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:51.971019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:51.971019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:51.971019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:51.971019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:51.971019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:51.971019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:51.971019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:51.971019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:51.971019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:51.971966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:51.973361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:51.973361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:51.973361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:51.973361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:51.973361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:51.973361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.973361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:51.976307 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:51.976616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:51.976756 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:51.976953 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:51.977271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:51.977526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.977955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:51.977955 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:51.977955 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:51.977955 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:51.977955 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:51.977955 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:51.977955 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.977955 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:51.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:51.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:51.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:51.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:51.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:51.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:51.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:51.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:51.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:51.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:51.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:51.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:51.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:51.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:51.979129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:51.979129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:51.979129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:51.979129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:51.979129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:51.979129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:51.979129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:51.979129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.979129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:51.979933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.979933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:51.979933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:51.979933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:51.979933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:51.979933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:51.979933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:51.979933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:51.979933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:51.979933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:51.979933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:51.980291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:51.980291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:51.980291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:51.980291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:51.980291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:51.980291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:51.980291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:51.980291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:51.980291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:51.980291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:51.980291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:51.980291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:51.980291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:51.980291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:51.980736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:51.980736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:51.980736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.980736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:51.981554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:51.982066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:51.982606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:51.982606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:51.982606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:51.982606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:52.009726 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:52.010175 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:52.010329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:52.010583 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:52.010966 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:52.011275 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:52.011796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:52.012338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:52.012338 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:52.012338 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:52.012338 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:52.012338 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:52.012338 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:52.012338 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:52.012338 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:52.013412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:52.013412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:52.013412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:52.013412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:52.013412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:52.013412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:52.013412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:52.013412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:52.013412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:52.013412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:52.013412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:52.013412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:52.013713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:52.013713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:52.013713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:52.013713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:52.013713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:52.013713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:52.013713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:52.013713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:52.013713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:52.013713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:52.013713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:52.013713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:52.013713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:52.014016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:52.014016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:52.017800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:52.017800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:52.017800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:52.017800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:52.017800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:52.017800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:52.017800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:52.017800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:52.017800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:52.017800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:52.017800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:52.017800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:52.017800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:52.017800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:52.017800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:52.017800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:52.018141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:52.018480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:52.018480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:52.018480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:52.034775 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:52.035026 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:52.035153 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:52.035942 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:52.086595 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:52.086815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:52.087273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:52.087273 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:52.087273 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:52.087273 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:52.087273 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:52.087273 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:52.087273 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:52.087273 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:52.088005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.052 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:52.088005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:52.088005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:52.088005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:52.088005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:52.088005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:52.088005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:52.088005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:52.088005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:52.088005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:52.088005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:52.088005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:52.088407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:52.088407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:52.088407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:52.088407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:52.088407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:52.088407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:52.088407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:52.088407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:52.088407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:52.088407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:52.088407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:52.088407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:52.088407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:52.088755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:52.088755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:52.089088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:52.089088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:52.089088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:52.089088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:52.089088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:52.089088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:52.089088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:52.089088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:52.089088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:52.089088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:52.089088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:52.089088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:52.089088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:52.089088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:52.089088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:52.089088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:52.089517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:52.089983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:52.089983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:52.089983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:56.894744 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:56.896055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:56.896055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:56.896731 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:56.898065 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:56.898492 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:56.898870 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:56.900128 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:56.900436 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:56.900687 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:56.900924 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:56.901462 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:56.902500 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:56.902820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:56.903856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:56.904124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.904915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:56.904915 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:56.904915 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:56.904915 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:56.904915 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:56.904915 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:56.904915 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.904915 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:56.906906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.906906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:56.906906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:56.906906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:56.906906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:56.906906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:56.906906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:56.906906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:56.906906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:56.906906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:56.906906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:56.906906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:56.907313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:56.907313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:56.907313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:56.907313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:56.907313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:56.907313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:56.907313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:56.907313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:56.907313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:56.907313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:56.907313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:56.907313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:56.907313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:56.907727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.907727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:56.907727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.907727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:56.907727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:56.907727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:56.907727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:56.907727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:56.907727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:56.907727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:56.907727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:56.907727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:56.907727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:56.907727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:56.907727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:56.908068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:56.908382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:56.908382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:56.908382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:56.908382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:56.908382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.908382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:56.959015 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:56.959625 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:56.959625 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:56.959625 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:40:56.960164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:56.961465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.961628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:56.961628 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:56.961628 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:56.961628 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:56.961628 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:56.961628 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:56.961628 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.961628 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:56.962708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.962708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:56.962708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:56.962708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:56.962708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:56.962708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:56.962708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:56.962708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:56.962708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:56.962708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:56.962708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:56.962708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:56.962708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:56.962708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:56.962708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:56.963891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:56.963891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:56.963891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:56.963891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:56.963891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:56.963891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:56.963891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:56.963891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.963891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:56.963891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.963891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:56.963891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:56.963891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:56.963891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:56.964157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:56.964157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:56.964157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:40:56.964157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:56.964157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:56.964157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:56.964157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:56.964157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:56.964157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:56.964157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:56.964157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:56.964157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:56.964157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:56.964360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:56.964360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:56.964360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:56.964360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:56.964360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:56.964360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:56.964360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:56.964360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:56.964360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:56.964360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:56.964360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.964360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:56.964633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.964633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:56.964633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:56.964633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:56.964633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:56.964633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:56.964633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:56.964633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:56.964633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:56.964633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:56.964633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:56.964633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:56.964633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:56.964633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:56.964633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:56.964633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:56.964875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:56.965189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:56.965189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:56.965189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:56.965189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:56.965189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:56.965189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.965189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:56.976786 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:56.977276 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:56.977467 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:56.977661 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:56.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:56.978692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.978888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:56.978888 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:56.978888 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:56.978888 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:56.978888 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:56.978888 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:56.978888 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.978888 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:56.979695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.979695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:40:56.979695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:56.979695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:56.979695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:56.979695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:56.979695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:56.979695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:40:56.979695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:56.979695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:40:56.979695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:56.979695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:40:56.979695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:56.979695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:40:56.979695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:40:56.980766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:56.980766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:40:56.980766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:56.980766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:40:56.980766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:40:56.980766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:56.980766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:40:56.980766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.980766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:40:56.980766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.980766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:56.980766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:56.980766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:56.980766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:56.981235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:56.981235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:56.981235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:56.981235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:56.981235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:56.981235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:56.981235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:56.981235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:56.981235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:56.981235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:56.981235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:56.981235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:56.981235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:56.981456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:56.981456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:56.981456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:56.981456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:56.981456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:56.981456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:56.981456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:56.981456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:56.981456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:56.981456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.981456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:56.982115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:56.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:56.983428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:56.983428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:56.983428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:56.983428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:57.010251 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:57.011636 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:57.011777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:57.011857 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:57.012291 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:57.012676 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:57.013064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:57.013653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:57.013653 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:57.013653 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:57.013653 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:57.013653 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:57.013653 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:57.013653 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:57.013653 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:57.014319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:57.014319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:57.014319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:57.014319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:57.014319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:57.014319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:57.014319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:57.014319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:57.014319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:57.014319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:57.014319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:57.014319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:57.014601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:57.014601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:57.014601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:57.014601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:57.014601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:57.014601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:57.014601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:57.014601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:57.014601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:57.014601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:57.014601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:57.014601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:57.014601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:57.014812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:57.014812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:57.015021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:57.015021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:57.015021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:57.015021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:57.015021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:57.015021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:57.015021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:57.015021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:57.015021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:57.015021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:57.015021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:57.015021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:57.015021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:57.015021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:57.015021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:57.015021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:57.015345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:57.015742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:57.015742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:57.015742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:40:57.035244 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:40:57.036441 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:40:57.036441 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:40:57.036634 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:40:57.085345 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:40:57.085800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:57.086310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:40:57.086310 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:57.086310 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:40:57.086310 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:40:57.086310 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:57.086310 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:40:57.086310 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:57.086310 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:40:57.087615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.050 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:57.087615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:40:57.087615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:57.087615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:40:57.087615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:57.087615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:57.087615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:57.087615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:57.087615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:57.087615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:57.087615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:40:57.087615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:57.088021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:40:57.088021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:57.088021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:40:57.088021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:57.088021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:40:57.088021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:40:57.088021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:57.088021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:40:57.088021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:57.088021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:40:57.088021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:40:57.088021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:57.088021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:40:57.088398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:57.088398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:40:57.088732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:57.088732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:40:57.088732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:40:57.088732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:40:57.088732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:40:57.088732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:40:57.088732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:57.088732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:40:57.088732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:40:57.088732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:57.088732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:40:57.088732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:40:57.088732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:40:57.088732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:40:57.088732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:40:57.088732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:40:57.089096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:40:57.089596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:40:57.089596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:40:57.089596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:01.895265 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:01.895892 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:01.895892 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:01.898632 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:01.898632 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:01.898632 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:01.898632 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:01.898632 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:01.898821 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:01.898821 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:01.899522 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:01.899822 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:01.900081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:01.900347 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:01.900661 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:01.900951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:01.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:01.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:01.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:01.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:01.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:01.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.902831 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:01.903687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.903687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:01.903687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:01.903687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:01.903687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:01.903687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:01.903687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:01.903687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:01.903687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:01.903687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:01.903687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:01.903687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:01.903960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:01.903960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:01.903960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:01.903960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:01.903960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:01.903960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:01.903960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:01.903960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:01.903960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:01.903960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:01.903960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:01.903960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:01.903960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:01.904231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.904231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:01.904395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.904395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:01.904395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:01.904395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:01.904395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:01.904395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:01.904395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:01.904395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:01.904395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:01.904395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:01.904395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:01.904395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:01.904395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:01.904395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:01.904395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:01.904395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:01.904743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:01.905033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:01.905033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.905033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:01.959590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:01.959930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:01.959982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:01.960064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:41:01.961901 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:01.962964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.967065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:01.967065 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:01.967065 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:01.967065 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:01.967065 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:01.967065 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:01.967065 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.967065 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:01.967792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.967792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:01.967792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:01.967792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:01.967792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:01.967792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:01.967792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:01.967792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:01.967792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:01.967792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:01.967792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:01.967792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:01.967792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:01.967792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:01.967792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:01.968062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:01.968062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:01.968062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:01.968062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:01.968062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:01.968062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:01.968062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:01.968062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.968062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:01.968477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.968477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:01.968477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:01.968477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:01.968477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:01.968477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:01.968477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:01.968477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:41:01.968477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:01.968477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:01.968477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:01.968477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:01.968741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:01.968741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:01.968741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:01.968741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:01.968741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:01.968741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:01.968741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:01.968741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:01.968741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:01.968741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:01.968741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:01.968741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:01.968741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:01.968741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:01.969260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:01.969260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:01.969260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.969260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:01.969260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.969260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:01.969260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:01.969260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:01.969260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:01.969260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:01.969260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:01.969260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:01.969260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:01.969260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:01.969515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:01.969817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:01.969817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:01.969817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:01.969817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:01.969817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:01.969817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:01.969817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:01.969817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:01.969817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:01.969817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:01.969817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.969817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:01.977252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:01.977652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:01.977812 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:01.977979 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:01.978977 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:01.979218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:01.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:01.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:01.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:01.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:01.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:01.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.981924 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:01.982547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.982547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:01.982547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:01.982547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:01.982547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:01.982547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:01.982547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:01.982547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:01.982547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:01.982547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:01.982547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:01.982547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:01.982547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:01.982547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:01.982547 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:01.982813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:01.982813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:01.982813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:01.982813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:01.982813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:01.982813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:01.982813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:01.982813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.982813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:01.983184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.983184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:01.983184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:01.983184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:01.983184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:01.983184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:01.983184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:01.983184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:01.983184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:01.983184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:01.983184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:01.983396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:01.983396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:01.983396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:01.983396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:01.983396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:01.983396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:01.983396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:01.983396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:01.983396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:01.983396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:01.983396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:01.983396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:01.983396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:01.983396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:01.983691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:01.983691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:01.983691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.983691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:01.983957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:01.984573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:01.984573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:01.984573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:01.984573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:02.011796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:02.012226 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:02.012347 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:02.012509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:02.012956 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:02.013241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:02.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:02.014288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:02.014288 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:02.014288 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:02.014288 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:02.014288 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:02.014288 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:02.014288 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:02.014288 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:02.015085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:02.015085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:02.015085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:02.015085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:02.015085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:02.015085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:02.015085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:02.015085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:02.015085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:02.015085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:02.015085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:02.015085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:02.015349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:02.015349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:02.015349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:02.015349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:02.015349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:02.015349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:02.015349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:02.015349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:02.015349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:02.015349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:02.015349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:02.015349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:02.015349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:02.015969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:02.015969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:02.015969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:02.015969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:02.015969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:02.015969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:02.015969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:02.015969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:02.015969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:02.015969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:02.015969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:02.015969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:02.015969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:02.015969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:02.015969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:02.016644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:02.017339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:02.017339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:02.017339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:02.017339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:02.017339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:02.017339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:02.035718 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:02.035959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:02.036048 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:02.036960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:02.090204 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:02.090204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:02.091736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:02.091736 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:02.091736 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:02.091736 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:02.091736 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:02.091736 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:02.091736 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:02.091736 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:02.091736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.055 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:02.091736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:02.091736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:02.091736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:02.091736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:02.091736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:02.091736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:02.092317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:02.092317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:02.092317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:02.092317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:02.092317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:02.092317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:02.092317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:02.092317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:02.092317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:02.092317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:02.092317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:02.092317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:02.092317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:02.092317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:02.092882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:02.092882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:02.092882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:02.092882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:02.092882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:02.092882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:02.092882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:02.092882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:02.092882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:02.092882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:02.092882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:02.092882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:02.092882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:02.092882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:02.093315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:02.093851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:02.093851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:02.093851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:02.093851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:02.093851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:02.093851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:02.093851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:02.093851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:02.093851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:06.895769 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:06.896301 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:06.896485 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:06.897603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:06.900086 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:06.900331 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:06.900509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:06.900698 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:06.900863 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:06.901015 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:06.901172 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:06.901339 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:06.901507 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:06.901683 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=10): DelStaticRoutesCommand(_result=None, lrouter=neutron-d032402b-d9bd-4edb-868c-286dafc01a55, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:06.902240 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=11): DelStaticRoutesCommand(_result=None, lrouter=neutron-fe6ed2e9-39b1-466d-9355-5182caaacad3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:06.902450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.906146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:06.906146 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:06.906146 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:06.906146 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:06.906146 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:06.906146 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:06.906146 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.906146 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:06.907054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.010 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.907054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:06.907054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:06.907054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:06.907054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:06.907054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:06.907054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:06.907054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:06.907054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:06.907054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:06.907054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:06.907054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:06.907505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:06.907505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:06.907505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:06.907505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:06.907505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:06.907505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:06.907505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:06.907505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:06.907505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:06.907505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:06.907505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:06.907505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:06.907505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:06.908017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.908017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:06.908017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.908017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:06.908017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:06.908017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:06.908017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:06.908017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:06.908017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:06.908017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:06.908017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:06.908017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:06.908017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:06.908017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:06.908017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:06.908442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:06.909046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:06.909046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:06.909046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:06.909046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:06.909046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.909046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:06.959945 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:06.960272 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:06.960365 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:06.960508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:41:06.960939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:06.961023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.961517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:06.961517 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:06.961517 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:06.961517 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:06.961517 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:06.961517 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:06.961517 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.961517 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:06.962310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.962310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:06.962310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:06.962310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:06.962310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:06.962310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:06.962310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:06.962310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:06.962310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:06.962310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:06.962310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:06.962310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:06.962310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:06.962310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:06.962310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:06.963019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:06.963019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:06.963019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:06.963019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:06.963019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:06.963019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:06.963019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:06.963019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.963019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:06.963174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.963174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:06.963174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:06.963174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:06.963174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:06.963174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:06.963174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:06.963174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:41:06.963174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:06.963174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:06.963174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:06.963174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:06.963378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:06.963378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:06.963378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:06.963378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:06.963378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:06.963378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:06.963378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:06.963378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:06.963378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:06.963378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:06.963378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:06.963378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:06.963378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:06.963378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:06.963608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:06.963608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:06.963608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.963608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:06.964274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.964274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:06.964274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:06.964274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:06.964274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:06.964274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:06.964274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:06.964274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:06.964274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:06.964274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:06.964274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:06.964274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:06.964274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:06.964274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:06.964274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:06.964274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:06.964595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:06.964908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:06.964908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:06.964908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:06.964908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:06.964908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:06.964908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.964908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:06.977679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:06.977925 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:06.978055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:06.978205 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:06.978582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:06.978765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.979261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:06.979261 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:06.979261 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:06.979261 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:06.979261 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:06.979261 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:06.979261 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.979261 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:06.979941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.979941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:06.979941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:06.979941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:06.979941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:06.979941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:06.979941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:06.979941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:06.979941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:06.979941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:06.979941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:06.979941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:06.979941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:06.979941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:06.979941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:06.980400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:06.980400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:06.980400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:06.980400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:06.980400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:06.980400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:06.980400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:06.980400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.980400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:06.981268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.981268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:06.981268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:06.981268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:06.981268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:06.981268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:06.981268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:06.981268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:06.981268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:06.981268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:06.981268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:06.982257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:06.982257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:06.982257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:06.982257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:06.982257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:06.982257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:06.982257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:06.982257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:06.982257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:06.982257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:06.982257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:06.982257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:06.982257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:06.982257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:06.983102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:06.983102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:06.983102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.983102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:06.983102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.983102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:06.983102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:06.983102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:06.983102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:06.983102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:06.983102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:06.983102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:06.983102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:06.983102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:06.983488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:06.983966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:06.983966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:06.983966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:06.983966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:06.983966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:06.983966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:06.983966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:06.983966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:06.983966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:06.983966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:06.983966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:07.012214 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:07.012588 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:07.012715 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:07.012893 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:07.013198 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:07.013389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:07.013699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:07.014459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:07.014459 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:07.014459 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:07.014459 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:07.014459 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:07.014459 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:07.014459 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:07.014459 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:07.015691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:07.015691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:07.015691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:07.015691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:07.015691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:07.015691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:07.015691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:07.015691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:07.015691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:07.015691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:07.015691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:07.015691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:07.015955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:07.015955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:07.015955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:07.015955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:07.015955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:07.015955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:07.015955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:07.015955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:07.015955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:07.015955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:07.015955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:07.015955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:07.015955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:07.016183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:07.016183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:07.016964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:07.016964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:07.016964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:07.016964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:07.016964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:07.016964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:07.016964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:07.016964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:07.016964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:07.016964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:07.016964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:07.016964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:07.016964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:07.016964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:07.016964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:07.016964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:07.017257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:07.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:07.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:07.017574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:07.036145 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:07.036693 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:07.036866 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:07.037886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:07.082185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:07.082375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:07.082983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:07.082983 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:07.082983 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:07.082983 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:07.082983 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:07.082983 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:07.082983 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:07.082983 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:07.084525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:07.084525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:07.084525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:07.084525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:07.084525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:07.084525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:07.084525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:07.084525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:07.084525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:07.084525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:07.084525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:07.084525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:07.084933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:07.084933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:07.084933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:07.084933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:07.084933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:07.084933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:07.084933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:07.084933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:07.084933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:07.084933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:07.084933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:07.084933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:07.084933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:07.085243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:07.085243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:07.085712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:07.085712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:07.085712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:07.085712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:07.085712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:07.085712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:07.085712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:07.085712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:07.085712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:07.085712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:07.085712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:07.085712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:07.085712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:07.085712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:07.085712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:07.085712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:07.086128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:07.086613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:07.086613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:07.086613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:11.896298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:11.896880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:11.896880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:11.897982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:11.898225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:11.899272 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-7f17481f-41b4-4cca-9c5f-2f20e531ae95, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:11.899272 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:11.899272 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:11.899272 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-578e1446-2419-4802-974b-dda1e3f7f9be, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:11.899272 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:11.899604 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:11.899813 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:11.900014 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:11.900227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.900992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:11.900992 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:11.900992 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:11.900992 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:11.900992 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:11.900992 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:11.900992 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.900992 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:11.901842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.901842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:11.901842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:11.901842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:11.901842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:11.901842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:11.901842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:11.901842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:11.901842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:11.901842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:11.901842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:11.901842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:11.902107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:11.902107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:11.902107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:11.902107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:11.902107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:11.902107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:11.902107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:11.902107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:11.902107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:11.902107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:11.902107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:11.902107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:11.902107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:11.902354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.902354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:11.902660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.902660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:11.902660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:11.902660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:11.902660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:11.902660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:11.902660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:11.902660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:11.902660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:11.902660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:11.902660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:11.902660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:11.902660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:11.902660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:11.902660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:11.902660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:11.902992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:11.903372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:11.903372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.903372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:11.960465 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:11.960964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:11.960964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:11.961080 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:41:11.961482 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:11.961681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.962195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:11.962195 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:11.962195 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:11.962195 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:11.962195 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:11.962195 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:11.962195 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.962195 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:11.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:11.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:11.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:11.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:11.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:11.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:11.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:11.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:11.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:11.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:11.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:11.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:11.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:11.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:11.963245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:11.963245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:11.963245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:11.963245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:11.963245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:11.963245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:11.963245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:11.963245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.963245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:11.963826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.963826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:11.963826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:11.963826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:11.963826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:11.963826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:11.963826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:11.963826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:41:11.963826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:11.963826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:11.963826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:11.963826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:11.964115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:11.964115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:11.964115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:11.964115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:11.964115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:11.964115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:11.964115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:11.964115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:11.964115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:11.964115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:11.964115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:11.964115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:11.964115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:11.964115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:11.964333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:11.964333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:11.964333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.964333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:11.964333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.964333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:11.964333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:11.964333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:11.964333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:11.964333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:11.964333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:11.964333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:11.964333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:11.964333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:11.964602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:11.964978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:11.964978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:11.964978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:11.964978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:11.964978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:11.964978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:11.964978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:11.964978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:11.964978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:11.964978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:11.964978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.964978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:11.978530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:11.978917 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:11.979015 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:11.979111 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:11.979891 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:11.979891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.980194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:11.980194 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:11.980194 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:11.980194 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:11.980194 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:11.980194 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:11.980194 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.980194 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:11.980849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.980849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:11.980849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:11.980849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:11.980849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:11.980849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:11.980849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:11.980849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:11.980849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:11.980849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:11.980849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:11.980849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:11.980849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:11.980849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:11.980849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:11.981251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:11.981251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:11.981251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:11.981251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:11.981251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:11.981251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:11.981251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:11.981251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.981251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:11.981416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.981416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:11.981416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:11.981416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:11.981416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:11.981416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:11.981416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:11.981416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:11.981416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:11.981416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:11.981416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:11.981702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:11.981702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:11.981702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:11.981702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:11.981702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:11.981702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:11.981702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:11.981702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:11.981702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:11.981702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:11.981702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:11.981702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:11.981702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:11.981702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:11.981929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:11.981929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:11.981929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.981929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:11.982371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:11.982744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:11.983049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:11.983049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:11.983049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:11.983049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:12.012850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:12.013440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:12.013534 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:12.013710 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:12.014055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:12.014194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:12.014561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:12.015116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:12.015116 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:12.015116 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:12.015116 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:12.015116 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:12.015116 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:12.015116 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:12.015116 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:12.016328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:12.016328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:12.016328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:12.016328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:12.016328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:12.016328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:12.016328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:12.016328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:12.016328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:12.016328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:12.016328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:12.016328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:12.016694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:12.016694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:12.016694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:12.016694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:12.016694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:12.016694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:12.016694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:12.016694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:12.016694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:12.016694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:12.016694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:12.016694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:12.016694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:12.018903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:12.018903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:12.018903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:12.018903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:12.018903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:12.018903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:12.018903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:12.018903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:12.018903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:12.018903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:12.018903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:12.018903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:12.018903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:12.018903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:12.018903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:12.019196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:12.019657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:12.019657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:12.019657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:12.019657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:12.019657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:12.019657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:12.036636 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:12.037052 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:12.037144 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:12.037766 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:12.070913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:12.071129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:12.071892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:12.071892 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:12.071892 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:12.071892 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:12.071892 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:12.071892 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:12.071892 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:12.071892 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:12.072564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:12.072564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:12.072564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:12.072564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:12.072564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:12.072564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:12.072564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:12.072564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:12.072564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:12.072564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:12.072564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:12.072564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:12.072881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:12.072881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:12.072881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:12.072881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:12.072881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:12.072881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:12.072881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:12.072881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:12.072881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:12.072881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:12.072881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:12.072881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:12.072881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:12.073491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:12.073491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:12.073491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:12.073491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:12.073491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:12.073491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:12.073491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:12.073491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:12.073491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:12.073491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:12.073491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:12.073491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:12.073491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:12.073491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:12.073491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:12.073827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:12.074367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:12.074367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:12.074367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:12.074367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:12.074367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:12.074367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:16.896869 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:16.897303 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:16.897303 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:16.897966 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:16.898098 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:16.898204 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:16.898301 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:16.898418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:16.898609 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:16.898754 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:16.898909 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:16.899058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.901114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:16.901114 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:16.901114 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:16.901114 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:16.901114 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:16.901114 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:16.901114 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.901114 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:16.901771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.901771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:16.901771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:16.901771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:16.901771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:16.901771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:16.901771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:16.901771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:16.901771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:16.901771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:16.901771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:16.901771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:16.902211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:16.902211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:16.902211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:16.902211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:16.902211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:16.902211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:16.902211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:16.902211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:16.902211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:16.902211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:16.902211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:16.902211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:16.902211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:16.902650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.902650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:16.902650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.902650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:16.902650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:16.902650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:16.902650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:16.902650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:16.902650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:16.902650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:16.902650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:16.902650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:16.902650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:16.902650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:16.902650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:16.902927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:16.903309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:16.903309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:16.903309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:16.903309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:16.903309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.903309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:16.963782 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:16.966866 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:16.966979 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:16.967083 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:41:16.967734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:16.968030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.970208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:16.970208 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:16.970208 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:16.970208 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:16.970208 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:16.970208 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:16.970208 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.970208 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:16.971420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.971420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:16.971420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:16.971420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:16.971420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:16.971420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:16.971420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:16.971420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:16.971420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:16.971420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:16.971420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:16.971420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:16.971420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:16.971420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:16.971420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:16.971960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:16.971960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:16.971960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:16.971960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:16.971960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:16.971960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:16.971960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:16.971960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.971960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:16.974559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.974559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:16.974559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:16.974559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:16.974559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:16.974559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:16.974559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:16.974559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:41:16.974559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:16.974559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:16.974559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:16.974559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:16.974931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:16.974931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:16.974931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:16.974931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:16.974931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:16.974931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:16.974931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:16.974931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:16.974931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:16.974931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:16.974931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:16.974931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:16.974931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:16.974931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:16.975288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:16.975288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:16.975288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.975288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:16.976704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.976704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:16.976704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:16.976704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:16.976704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:16.976704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:16.976704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:16.976704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:16.976704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:16.976704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:16.976704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:16.976704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:16.976704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:16.976704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:16.976704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:16.976704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:16.977177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:16.977679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:16.977679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:16.977679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:16.977679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:16.977679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:16.977679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.977679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:16.978946 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:16.979251 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:16.979352 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:16.979479 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:16.979925 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:16.983560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:16.984076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:16.984076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:16.984076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:16.984076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:16.984076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:16.984076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:16.984076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:16.984076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:16.984076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:16.984076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:16.984076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:16.984076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:16.984076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:16.984076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:16.984076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.984076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:16.984494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.984494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:16.984494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:16.984494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:16.984494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:16.984494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:16.984494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:16.984494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:16.984494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:16.984494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:16.984494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:16.984822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:16.984822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:16.984822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:16.984822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:16.984822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:16.984822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:16.984822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:16.984822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:16.984822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:16.984822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:16.984822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:16.984822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:16.984822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:16.984822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:16.985150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:16.985150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:16.985150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.985150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:16.985150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.985150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:16.985150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:16.985150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:16.985150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:16.985150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:16.985150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:16.985150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:16.985150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:16.985150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:16.985530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:16.986019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:16.986019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:16.986019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:16.986019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:16.986019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:16.986019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:16.986019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:16.986019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:16.986019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:16.986019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:16.986019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:17.013529 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:17.014114 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:17.014264 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:17.014467 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:17.014870 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:17.015108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:17.015492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:17.016079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:17.016079 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:17.016079 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:17.016079 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:17.016079 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:17.016079 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:17.016079 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:17.016079 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:17.016864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:17.016864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:17.016864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:17.016864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:17.016864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:17.016864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:17.016864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:17.016864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:17.016864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:17.016864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:17.016864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:17.016864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:17.017280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:17.017280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:17.017280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:17.017280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:17.017280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:17.017280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:17.017280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:17.017280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:17.017280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:17.017280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:17.017280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:17.017280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:17.017280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:17.017680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:17.017680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:17.017747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:17.017747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:17.017747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:17.017747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:17.017747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:17.017747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:17.017747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:17.017747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:17.017747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:17.017747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:17.017747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:17.017747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:17.017747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:17.017747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:17.017747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:17.017747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:17.018223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:17.018786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:17.018786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:17.018786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:17.037151 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:17.037527 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:17.037647 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:17.038437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:17.076173 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:17.076360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:17.077166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:17.077166 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:17.077166 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:17.077166 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:17.077166 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:17.077166 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:17.077166 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:17.077166 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:17.077899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:17.077899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:17.077899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:17.077899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:17.077899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:17.077899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:17.077899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:17.077899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:17.077899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:17.077899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:17.077899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:17.077899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:17.078287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:17.078287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:17.078287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:17.078287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:17.078287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:17.078287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:17.078287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:17.078287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:17.078287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:17.078287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:17.078287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:17.078287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:17.078287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:17.078589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:17.078589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:17.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:17.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:17.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:17.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:17.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:17.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:17.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:17.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:17.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:17.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:17.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:17.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:17.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:17.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:17.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:17.078642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:17.078987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:17.079398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:17.079398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:17.079398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:21.897194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:21.898901 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:21.898901 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:21.899284 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:21.899575 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:21.899733 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:21.899907 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:21.900083 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:21.900256 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:21.900431 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:21.900618 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:21.900779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:21.900946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.901598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:21.901598 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:21.901598 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:21.901598 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:21.901598 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:21.901598 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:21.901598 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.901598 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:21.903481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.903481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:21.903481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:21.903481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:21.903481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:21.903481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:21.903481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:21.903481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:21.903481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:21.903481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:21.903481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:21.903481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:21.903737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:21.903737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:21.903737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:21.903737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:21.903737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:21.903737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:21.903737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:21.903737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:21.903737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:21.903737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:21.903737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:21.903737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:21.903737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:21.903961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.903961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:21.903961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.903961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:21.903961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:21.903961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:21.903961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:21.903961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:21.903961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:21.903961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:21.903961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:21.903961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:21.903961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:21.903961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:21.903961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:21.904204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:21.904533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:21.904533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:21.904533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:21.904533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:21.904533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.904533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:21.969017 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:21.970466 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:21.970466 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:21.970466 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:41:21.970783 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:21.970961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:21.974450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:21.975006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:21.975006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:21.975006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:21.975006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:21.975006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:21.975006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:21.975006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:21.975006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:21.975006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:21.975006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:21.975006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:21.975006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:21.975006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.975006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:21.975006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.975513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:21.975513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:21.975513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:21.975513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:21.975513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:21.975513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:21.975513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:41:21.975513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:21.975513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:21.975513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:21.975513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:21.975513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:21.975513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:21.975784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:21.975784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:21.975784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:21.975784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:21.975784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:21.975784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:21.975784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:21.975784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:21.975784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:21.975784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:21.975784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:21.975784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:21.975784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:21.975784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:21.976322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.976322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:21.976322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.976322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:21.976322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:21.976322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:21.976322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:21.976322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:21.976322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:21.976322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:21.976322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:21.976322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:21.976322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:21.976322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:21.976322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:21.976863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:21.977442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:21.977442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:21.977442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:21.977442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:21.977442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:21.977442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:21.977442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:21.977442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.977442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:21.979400 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:21.979758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:21.979999 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:21.980263 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:21.981253 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:21.981575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:21.984762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:21.985479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:21.985479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:21.985479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:21.985479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:21.985479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:21.985479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:21.985479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:21.985479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:21.985479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:21.985479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:21.985479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:21.985479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:21.985479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.985479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:21.985479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.986176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:21.986176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:21.986176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:21.986176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:21.986176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:21.986176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:21.986176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:21.986176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:21.986176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:21.986176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:21.986176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:21.986176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:21.986176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:21.986533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:21.986533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:21.986533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:21.986533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:21.986533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:21.986533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:21.986533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:21.986533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:21.986533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:21.986533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:21.986533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:21.986533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:21.986533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:21.987565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.987565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:21.987565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.987565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:21.987565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:21.987565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:21.987565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:21.987565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:21.987565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:21.987565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:21.987565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:21.987565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:21.987565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:21.987565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:21.987565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:21.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:21.988719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:21.988719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:21.988719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:21.988719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:21.988719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:21.988719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:21.988719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:21.988719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:22.014246 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:22.015237 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:22.015896 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:22.016140 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:22.019047 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:22.019358 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:22.019699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:22.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:22.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:22.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:22.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:22.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:22.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:22.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:22.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:22.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:22.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:22.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:22.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:22.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:22.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:22.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:22.022752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:22.022752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:22.022752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:22.022752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:22.022752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:22.022752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:22.022752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:22.022752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:22.022752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:22.022752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:22.022752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:22.022752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:22.022752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:22.022752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:22.023072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:22.023072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:22.023072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:22.023072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:22.023072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:22.023072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:22.023072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:22.023072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:22.023072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:22.023072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:22.023072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:22.023072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:22.023072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:22.023072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:22.023392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:22.023780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:22.023780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:22.023780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:22.023780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:22.023780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:22.023780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:22.023780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:22.023780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:22.023780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:22.023780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:22.037597 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:22.038855 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:22.039168 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:22.041816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:22.049486 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=239 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:41:22.082951 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:22.083211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:22.083940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:22.083940 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:22.083940 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:22.083940 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:22.083940 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:22.083940 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:22.083940 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:22.083940 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:22.084673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:22.084673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:22.084673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:22.084673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:22.084673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:22.084673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:22.084673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:22.084673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:22.084673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:22.084673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:22.084673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:22.084673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:22.085017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:22.085017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:22.085017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:22.085017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:22.085017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:22.085017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:22.085017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:22.085017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:22.085017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:22.085017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:22.085017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:22.085017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:22.085017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:22.085317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:22.085317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:22.085376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:22.085376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:22.085376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:22.085376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:22.085376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:22.085376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:22.085376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:22.085376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:22.085376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:22.085376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:22.085376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:22.085376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:22.085376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:22.085376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:22.085376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:22.085376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:22.085799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:22.086222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:22.086222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:22.086222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:26.898039 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:26.898498 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:26.898661 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:26.900046 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:26.900330 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:26.900536 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:26.900733 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-85b356ec-0156-421d-b315-30fdb7afebe3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:26.900868 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:26.900970 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:26.901074 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:26.901192 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:26.901308 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=8): DelStaticRoutesCommand(_result=None, lrouter=neutron-7fafc175-efec-45d7-b668-d2b9f708b85f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:26.901410 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=9): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:26.901580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.902286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:26.902286 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:26.902286 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:26.902286 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:26.902286 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:26.902286 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:26.902286 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.902286 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:26.903329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.903329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:26.903329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:26.903329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:26.903329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:26.903329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:26.903329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:26.903329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:26.903329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:26.903329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:26.903329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:26.903329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:26.903815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:26.903815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:26.903815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:26.903815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:26.903815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:26.903815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:26.903815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:26.903815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:26.903815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:26.903815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:26.903815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:26.903815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:26.903815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:26.904737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.904737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:26.904737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.904737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:26.904737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:26.904737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:26.904737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:26.904737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:26.904737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:26.904737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:26.904737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:26.904737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:26.904737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:26.904737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:26.904737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:26.905308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:26.906041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:26.906041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:26.906041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:26.906041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:26.906041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.906041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:26.970132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:26.970587 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:26.970718 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:26.970868 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:41:26.971387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:26.971645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:26.973709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:26.974395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:26.974395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:26.974395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:26.974395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:26.974395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:26.974395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:26.974395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:26.974395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:26.974395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:26.974395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:26.974395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:26.974395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:26.974395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.974395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:26.974395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.974779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:26.974779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:26.974779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:26.974779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:26.974779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:26.974779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:26.974779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:41:26.974779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:26.974779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:26.974779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:26.974779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:26.974779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:26.974779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:26.975233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:26.975233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:26.975233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:26.975233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:26.975233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:26.975233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:26.975233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:26.975233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:26.975233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:26.975233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:26.975233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:26.975233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:26.975233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:26.975233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:26.975583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.975583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:26.975583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.975583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:26.975583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:26.975583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:26.975583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:26.975583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:26.975583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:26.975583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:26.975583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:26.975583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:26.975583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:26.975583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:26.975583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:26.975998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:26.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:26.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:26.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:26.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:26.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:26.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:26.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:26.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.976652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:26.979786 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:26.980038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:26.980167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:26.980327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:26.981224 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:26.981224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.981733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:26.981733 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:26.981733 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:26.981733 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:26.981733 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:26.981733 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:26.981733 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.981733 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:26.982725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.982725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:26.982725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:26.982725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:26.982725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:26.982725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:26.982725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:26.982725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:26.982725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:26.982725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:26.982725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:26.982725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:26.982725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:26.982725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:26.982725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:26.982995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:26.982995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:26.982995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:26.982995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:26.982995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:26.982995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:26.982995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:26.982995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.982995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:26.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:26.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:26.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:26.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:26.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:26.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:26.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:26.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:26.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:26.983656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:26.983904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:26.983904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:26.983904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:26.983904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:26.983904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:26.983904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:26.983904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:26.983904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:26.983904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:26.983904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:26.983904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:26.983904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:26.983904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:26.983904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:26.984115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:26.984115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:26.984115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.984115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:26.984900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:26.985237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:26.985818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:26.985818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:26.985818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:26.985818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:27.015276 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:27.015555 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:27.015657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:27.015807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:27.016349 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:27.016853 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:27.017210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:27.017881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:27.017881 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:27.017881 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:27.017881 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:27.017881 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:27.017881 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:27.017881 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:27.017881 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:27.018679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:27.018679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:27.018679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:27.018679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:27.018679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:27.018679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:27.018679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:27.018679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:27.018679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:27.018679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:27.018679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:27.018679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:27.019054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:27.019054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:27.019054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:27.019054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:27.019054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:27.019054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:27.019054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:27.019054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:27.019054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:27.019054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:27.019054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:27.019054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:27.019054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:27.019468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:27.019468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:27.019808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:27.019808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:27.019808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:27.019808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:27.019808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:27.019808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:27.019808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:27.019808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:27.019808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:27.019808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:27.019808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:27.019808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:27.019808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:27.019808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:27.019808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:27.019808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:27.023205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:27.024284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:27.024284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:27.024284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:27.038885 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:27.039138 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:27.039218 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:27.040030 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:27.086798 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:27.086798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:27.087056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:27.087056 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:27.087056 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:27.087056 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:27.087056 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:27.087056 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:27.087056 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:27.087056 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:27.088137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:27.088137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:27.088137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:27.088137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:27.088137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:27.088137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:27.088137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:27.088137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:27.088137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:27.088137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:27.088137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:27.088137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:27.088561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:27.088561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:27.088561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:27.088561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:27.088561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:27.088561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:27.088561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:27.088561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:27.088561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:27.088561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:27.088561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:27.088561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:27.088561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:27.088920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:27.088920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:27.089648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:27.089648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:27.089648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:27.089648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:27.089648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:27.089648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:27.089648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:27.089648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:27.089648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:27.089648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:27.089648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:27.089648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:27.089648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:27.089648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:27.089648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:27.089648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:27.090248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:27.090749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:27.090749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:27.090749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:31.898891 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:31.899362 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:31.899362 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:31.900344 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:31.900594 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:31.900779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:31.900942 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:31.901101 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:31.901686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:31.901852 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:31.902009 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:31.902169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.903700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:31.903700 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:31.903700 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:31.903700 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:31.903700 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:31.903700 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:31.903700 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.903700 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:31.904466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.904466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:31.904466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:31.904466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:31.904466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:31.904466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:31.904466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:31.904466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:31.904466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:31.904466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:31.904466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:31.904466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:31.904704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:31.904704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:31.904704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:31.904704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:31.904704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:31.904704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:31.904704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:31.904704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:31.904704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:31.904704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:31.904704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:31.904704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:31.904704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:31.904942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.904942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:31.905291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.905291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:31.905291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:31.905291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:31.905291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:31.905291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:31.905291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:31.905291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:31.905291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:31.905291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:31.905291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:31.905291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:31.905291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:31.905291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:31.905291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:31.905291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:31.905625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:31.905917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:31.905917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.905917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:31.970623 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:31.972575 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:31.972575 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:31.972758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:41:31.973256 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:31.973256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.973916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:31.973916 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:31.973916 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:31.973916 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:31.973916 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:31.973916 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:31.973916 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.973916 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:31.974995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.974995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:31.974995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:31.974995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:31.974995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:31.974995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:31.974995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:31.974995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:31.974995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:31.974995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:31.974995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:31.974995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:31.974995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:31.974995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:31.974995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:31.975933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:31.975933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:31.975933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:31.975933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:31.975933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:31.975933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:31.975933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:31.975933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.975933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:31.975933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.975933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:31.975933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:31.975933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:31.975933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:31.976256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:31.976256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:31.976256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:41:31.976256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:31.976256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:31.976256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:31.976256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:31.976256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:31.976256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:31.976256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:31.976256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:31.976256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:31.976256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:31.976584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:31.976584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:31.976584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:31.976584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:31.976584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:31.976584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:31.976584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:31.976584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:31.976584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:31.976584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:31.976584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.976584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:31.976826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.976826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:31.976826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:31.976826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:31.976826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:31.976826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:31.976826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:31.976826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:31.976826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:31.976826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:31.976826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:31.976826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:31.976826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:31.976826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:31.976826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:31.976826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:31.977079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:31.977347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:31.977347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:31.977347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:31.977347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:31.977347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:31.977347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.977347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:31.980065 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:31.980784 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:31.980784 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:31.980784 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:31.981228 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:31.981451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.982915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:31.982915 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:31.982915 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:31.982915 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:31.982915 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:31.982915 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:31.982915 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.982915 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:31.983927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.983927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:31.983927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:31.983927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:31.983927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:31.983927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:31.983927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:31.983927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:31.983927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:31.983927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:31.983927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:31.983927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:31.983927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:31.983927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:31.983927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:31.984750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:31.984750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:31.984750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:31.984750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:31.984750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:31.984750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:31.984750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:31.984750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.984750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:31.986749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.986749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:31.986749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:31.986749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:31.986749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:31.986749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:31.986749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:31.986749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:31.986749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:31.986749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:31.986749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:31.987160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:31.987160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:31.987160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:31.987160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:31.987160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:31.987160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:31.987160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:31.987160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:31.987160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:31.987160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:31.987160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:31.987160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:31.987160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:31.987160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:31.987394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:31.987394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:31.987394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.987394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:31.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:31.989021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:31.989337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:31.989337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:31.989337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:31.989337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:32.015948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:32.016182 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:32.016296 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:32.016449 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:32.016960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:32.017232 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:32.017838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:32.018106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:32.018106 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:32.018106 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:32.018106 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:32.018106 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:32.018106 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:32.018106 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:32.018106 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:32.019074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:32.019074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:32.019074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:32.019074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:32.019074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:32.019074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:32.019074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:32.019074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:32.019074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:32.019074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:32.019074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:32.019074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:32.019385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:32.019385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:32.019385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:32.019385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:32.019385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:32.019385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:32.019385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:32.019385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:32.019385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:32.019385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:32.019385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:32.019385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:32.019385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:32.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:32.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:32.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:32.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:32.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:32.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:32.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:32.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:32.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:32.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:32.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:32.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:32.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:32.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:32.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:32.020522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:32.021638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:32.021638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:32.021638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:32.021638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:32.021638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:32.021638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:32.039598 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:32.039982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:32.040149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:32.041043 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:32.079766 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:32.079970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:32.080873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:32.080873 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:32.080873 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:32.080873 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:32.080873 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:32.080873 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:32.080873 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:32.080873 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:32.081697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:32.081697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:32.081697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:32.081697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:32.081697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:32.081697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:32.081697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:32.081697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:32.081697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:32.081697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:32.081697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:32.081697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:32.082051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:32.082051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:32.082051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:32.082051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:32.082051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:32.082051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:32.082051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:32.082051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:32.082051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:32.082051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:32.082051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:32.082051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:32.082051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:32.082642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:32.082642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:32.082968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:32.082968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:32.082968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:32.082968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:32.082968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:32.082968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:32.082968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:32.082968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:32.082968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:32.082968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:32.082968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:32.082968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:32.082968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:32.082968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:32.082968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:32.082968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:32.083737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:32.084612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:32.084612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:32.084612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:36.899288 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:36.902863 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:36.902863 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:36.903271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:36.903271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:36.903623 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:36.903761 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:36.903844 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:36.904012 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:36.904162 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:36.904314 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:36.904480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.905921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:36.905921 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:36.905921 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:36.905921 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:36.905921 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:36.905921 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:36.905921 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.905921 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:36.906229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.906229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:36.906229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:36.906229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:36.906229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:36.906229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:36.906229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:36.906229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:36.906229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:36.906229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:36.906229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:36.906229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:36.906670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:36.906670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:36.906670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:36.906670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:36.906670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:36.906670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:36.906670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:36.906670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:36.906670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:36.906670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:36.906670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:36.906670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:36.906670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:36.907071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.907071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:36.909553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.909553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:36.909553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:36.909553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:36.909553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:36.909553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:36.909553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:36.909553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:36.909553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:36.909553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:36.909553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:36.909553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:36.909553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:36.909553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:36.909553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:36.909553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:36.909964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:36.910313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:36.910313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.910313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:36.971298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:36.971566 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:36.971665 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:36.971811 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:41:36.972038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:36.972195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.972604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:36.972604 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:36.972604 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:36.972604 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:36.972604 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:36.972604 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:36.972604 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.972604 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:36.973101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.973101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:36.973101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:36.973101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:36.973101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:36.973101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:36.973101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:36.973101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:36.973101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:36.973101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:36.973101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:36.973101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:36.973101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:36.973101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:36.973101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:36.973595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:36.973595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:36.973595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:36.973595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:36.973595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:36.973595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:36.973595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:36.973595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.973595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:36.973895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.973895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:36.973895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:36.973895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:36.973895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:36.973895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:36.973895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:36.973895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:41:36.973895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:36.973895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:36.973895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:36.973895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:36.974271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:36.974271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:36.974271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:36.974271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:36.974271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:36.974271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:36.974271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:36.974271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:36.974271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:36.974271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:36.974271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:36.974271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:36.974271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:36.974271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:36.974678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:36.974678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:36.974678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.974678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:36.974678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.974678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:36.974678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:36.974678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:36.974678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:36.974678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:36.974678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:36.974678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:36.974678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:36.974678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:36.975134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:36.975637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:36.975637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:36.975637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:36.975637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:36.975637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:36.975637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:36.975637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:36.975637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:36.975637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:36.975637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:36.975637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.975637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:36.981925 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:36.981925 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:36.981925 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:36.981925 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:36.981925 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:36.981925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:36.982284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:36.982914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:36.982914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:36.982914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:36.982914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:36.982914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:36.982914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:36.982914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:36.982914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:36.982914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:36.982914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:36.982914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:36.982914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:36.982914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.982914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:36.982914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.985652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:36.985652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:36.985652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:36.985652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:36.985652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:36.985652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:36.985652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:36.985652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:36.985652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:36.985652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:36.985652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:36.985652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:36.985652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:36.986095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:36.986095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:36.986095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:36.986095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:36.986095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:36.986095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:36.986095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:36.986095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:36.986095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:36.986095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:36.986095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:36.986095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:36.986095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:36.986468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.986468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:36.986468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.986468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:36.986468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:36.986468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:36.986468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:36.986468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:36.986468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:36.986468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:36.986468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:36.986468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:36.986468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:36.986468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:36.986468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:36.986911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:36.987443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:36.987443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:36.987443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:36.987443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:36.987443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:36.987443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:36.987443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:36.987443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:37.016368 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:37.017108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:37.017238 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:37.017506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:37.018130 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:37.018516 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:37.018750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:37.019414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:37.019414 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:37.019414 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:37.019414 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:37.019414 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:37.019414 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:37.019414 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:37.019414 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:37.020944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:37.020944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:37.020944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:37.020944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:37.020944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:37.020944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:37.020944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:37.020944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:37.020944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:37.020944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:37.020944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:37.020944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:37.021712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:37.021712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:37.021712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:37.021712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:37.021712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:37.021712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:37.021712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:37.021712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:37.021712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:37.021712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:37.021712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:37.021712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:37.021712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:37.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:37.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:37.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:37.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:37.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:37.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:37.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:37.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:37.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:37.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:37.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:37.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:37.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:37.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:37.021959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:37.022309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:37.022709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:37.022709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:37.022709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:37.022709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:37.022709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:37.022709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:37.040144 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:37.041453 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:37.041608 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:37.042221 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:37.090499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:37.090718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:37.091287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:37.091287 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:37.091287 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:37.091287 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:37.091287 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:37.091287 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:37.091287 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:37.091287 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:37.092328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.050 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:37.092328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:37.092328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:37.092328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:37.092328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:37.092328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:37.092328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:37.092328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:37.092328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:37.092328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:37.092328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:37.092328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:37.092842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:37.092842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:37.092842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:37.092842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:37.092842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:37.092842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:37.092842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:37.092842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:37.092842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:37.092842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:37.092842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:37.092842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:37.092842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:37.093238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:37.093238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:37.093421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:37.093421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:37.093421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:37.093421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:37.093421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:37.093421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:37.093421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:37.093421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:37.093421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:37.093421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:37.093421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:37.093421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:37.093421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:37.093421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:37.093421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:37.093421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:37.093863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:37.094195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:37.094195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:37.094195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:41.902026 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:41.902567 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:41.902833 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:41.903837 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:41.904034 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:41.904167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:41.904342 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:41.904399 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:41.904523 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:41.904641 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:41.904753 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:41.904858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.905336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:41.905336 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:41.905336 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:41.905336 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:41.905336 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:41.905336 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:41.905336 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.905336 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:41.906130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.906130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:41.906130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:41.906130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:41.906130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:41.906130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:41.906130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:41.906130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:41.906130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:41.906130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:41.906130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:41.906130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:41.906360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:41.906360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:41.906360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:41.906360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:41.906360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:41.906360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:41.906360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:41.906360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:41.906360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:41.906360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:41.906360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:41.906360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:41.906360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:41.906588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.906588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:41.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:41.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:41.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:41.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:41.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:41.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:41.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:41.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:41.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:41.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:41.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:41.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:41.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:41.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:41.908573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:41.908968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:41.909262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:41.909262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.909262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:41.971759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:41.972018 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:41.972108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:41.972196 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:41:41.972734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:41.972908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.973358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:41.973358 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:41.973358 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:41.973358 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:41.973358 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:41.973358 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:41.973358 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.973358 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:41.974008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.974008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:41.974008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:41.974008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:41.974008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:41.974008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:41.974008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:41.974008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:41.974008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:41.974008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:41.974008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:41.974008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:41.974008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:41.974008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:41.974008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:41.974261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:41.974261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:41.974261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:41.974261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:41.974261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:41.974261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:41.974261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:41.974261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.974261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:41.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:41.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:41.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:41.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:41.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:41.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:41.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:41:41.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:41.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:41.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:41.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:41.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:41.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:41.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:41.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:41.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:41.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:41.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:41.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:41.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:41.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:41.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:41.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:41.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:41.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:41.975137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:41.975137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:41.975137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.975137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:41.975289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.975289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:41.975289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:41.975289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:41.975289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:41.975289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:41.975289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:41.975289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:41.975289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:41.975289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:41.975289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:41.975289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:41.975289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:41.975289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:41.975289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:41.975289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:41.975638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:41.975917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:41.975917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:41.975917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:41.975917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:41.975917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:41.975917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.975917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:41.980833 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:41.981133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:41.981253 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:41.981391 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:41.981811 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:41.982023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.982422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:41.982422 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:41.982422 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:41.982422 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:41.982422 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:41.982422 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:41.982422 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.982422 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:41.983178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.983178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:41.983178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:41.983178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:41.983178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:41.983178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:41.983178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:41.983178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:41.983178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:41.983178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:41.983178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:41.983178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:41.983178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:41.983178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:41.983178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:41.983650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:41.983650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:41.983650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:41.983650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:41.983650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:41.983650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:41.983650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:41.983650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.983650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:41.984260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.984260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:41.984260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:41.984260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:41.984260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:41.984260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:41.984260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:41.984260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:41.984260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:41.984260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:41.984260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:41.984640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:41.984640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:41.984640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:41.984640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:41.984640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:41.984640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:41.984640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:41.984640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:41.984640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:41.984640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:41.984640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:41.984640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:41.984640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:41.984640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:41.985034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:41.985034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:41.985034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.985034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:41.985621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:41.986112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:41.986646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:41.986646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:41.986646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:41.986646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:42.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:42.017599 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:42.017682 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:42.017832 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:42.018378 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:42.018755 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:42.019111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:42.019826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:42.019826 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:42.019826 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:42.019826 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:42.019826 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:42.019826 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:42.019826 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:42.019826 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:42.020556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:42.020556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:42.020556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:42.020556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:42.020556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:42.020556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:42.020556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:42.020556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:42.020556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:42.020556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:42.020556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:42.020556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:42.020912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:42.020912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:42.020912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:42.020912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:42.020912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:42.020912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:42.020912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:42.020912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:42.020912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:42.020912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:42.020912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:42.020912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:42.020912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:42.021296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:42.021296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:42.021365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:42.021365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:42.021365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:42.021365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:42.021365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:42.021365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:42.021365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:42.021365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:42.021365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:42.021365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:42.021365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:42.021365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:42.021365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:42.021365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:42.021365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:42.021365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:42.021856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:42.022372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:42.022372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:42.022372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:42.040826 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:42.041105 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:42.041249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:42.041911 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:42.080574 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:42.080820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:42.081270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:42.081270 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:42.081270 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:42.081270 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:42.081270 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:42.081270 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:42.081270 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:42.081270 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:42.081943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:42.081943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:42.081943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:42.081943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:42.081943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:42.081943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:42.081943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:42.081943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:42.081943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:42.081943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:42.081943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:42.081943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:42.082187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:42.082187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:42.082187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:42.082187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:42.082187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:42.082187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:42.082187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:42.082187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:42.082187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:42.082187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:42.082187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:42.082187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:42.082187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:42.082587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:42.082587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:42.082708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:42.082708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:42.082708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:42.082708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:42.082708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:42.082708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:42.082708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:42.082708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:42.082708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:42.082708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:42.082708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:42.082708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:42.082708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:42.082708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:42.082708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:42.082708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:42.083145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:42.083679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:42.083679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:42.083679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:46.904573 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:46.904573 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:46.904573 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:46.904573 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:46.904573 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:46.905233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:46.905233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:46.905233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:46.905233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:46.905233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:46.905397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:46.905397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.905683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:46.905683 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:46.905683 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:46.905683 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:46.905683 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:46.905683 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:46.905683 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.905683 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:46.906493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.906493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:46.906493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:46.906493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:46.906493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:46.906493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:46.906493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:46.906493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:46.906493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:46.906493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:46.906493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:46.906493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:46.906787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:46.906787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:46.906787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:46.906787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:46.906787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:46.906787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:46.906787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:46.906787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:46.906787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:46.906787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:46.906787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:46.906787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:46.906787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:46.907053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.907053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:46.907267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.907267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:46.907267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:46.907267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:46.907267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:46.907267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:46.907267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:46.907267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:46.907267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:46.907267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:46.907267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:46.907267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:46.907267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:46.907267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:46.907267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:46.907267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:46.907593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:46.907914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:46.907914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.907914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:46.972187 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:46.972508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:46.972645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:46.972759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:41:46.975948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:46.976596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.977136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:46.977136 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:46.977136 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:46.977136 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:46.977136 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:46.977136 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:46.977136 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.977136 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:46.981930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.981930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:46.981930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:46.981930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:46.981930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:46.981930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:46.981930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:46.981930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:46.981930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:46.981930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:46.981930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:46.981930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:46.981930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:46.981930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:46.981930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:46.982721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:46.982721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:46.982721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:46.982721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:46.982721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:46.982721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:46.982721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:46.982721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.982721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:46.982996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.009 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.982996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:46.982996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:46.982996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:46.982996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:46.982996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:46.982996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:46.982996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:41:46.982996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:46.982996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:46.982996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:46.982996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:46.983440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:46.983440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:46.983440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:46.983440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:46.983440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:46.983440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:46.983440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:46.983440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:46.983440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:46.983440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:46.983440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:46.983440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:46.983440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:46.983440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:46.983958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:46.983958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:46.983958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.983958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:46.983958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:46.985131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.985131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:46.985131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:46.985131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:46.985131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:46.985131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:46.985131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:46.985131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:46.985131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:46.985131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:46.985131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:46.985131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:46.985131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:46.985131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:46.985131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:46.985131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:46.985588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:46.985953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:46.985953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:46.985953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:46.985953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:46.985953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:46.985953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.985953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:46.989435 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:46.989560 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:46.989824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:46.990342 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:46.990606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.991096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:46.991096 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:46.991096 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:46.991096 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:46.991096 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:46.991096 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:46.991096 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.991096 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:46.991816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.991816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:46.991816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:46.991816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:46.991816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:46.991816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:46.991816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:46.991816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:46.991816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:46.991816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:46.991816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:46.991816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:46.991816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:46.991816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:46.991816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:46.992123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:46.992123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:46.992123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:46.992123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:46.992123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:46.992123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:46.992123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:46.992123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.992123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:46.992412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.992412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:46.992412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:46.992412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:46.992412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:46.992412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:46.992412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:46.992412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:46.992412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:46.992412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:46.992412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:46.992818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:46.992818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:46.992818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:46.992818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:46.992818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:46.992818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:46.992818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:46.992818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:46.992818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:46.992818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:46.992818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:46.992818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:46.992818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:46.992818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:46.993228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:46.993228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:46.993228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.993228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:46.993228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.993228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:46.993228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:46.993228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:46.993228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:46.993228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:46.993228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:46.993228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:46.993228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:46.993228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:46.993493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:46.994008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:46.994008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:46.994008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:46.994008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:46.994008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:46.994008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:46.994008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:46.994008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:46.994008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:46.994008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:46.994008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:47.017756 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:47.021042 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:47.021042 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:47.021042 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:47.021643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:47.022073 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:47.022303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:47.023001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:47.023001 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:47.023001 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:47.023001 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:47.023001 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:47.023001 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:47.023001 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:47.023001 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:47.024026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:47.024026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:47.024026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:47.024026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:47.024026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:47.024026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:47.024026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:47.024026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:47.024026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:47.024026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:47.024026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:47.024026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:47.024701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:47.024701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:47.024701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:47.024701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:47.024701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:47.024701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:47.024701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:47.024701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:47.024701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:47.024701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:47.024701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:47.024701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:47.024701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:47.024984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:47.024984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:47.025052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:47.025052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:47.025052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:47.025052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:47.025052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:47.025052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:47.025052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:47.025052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:47.025052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:47.025052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:47.025052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:47.025052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:47.025052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:47.025052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:47.025052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:47.025052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:47.025382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:47.025865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:47.025865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:47.025865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:47.041618 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:47.043815 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:47.043964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:47.044810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:47.103289 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:47.103681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:47.104301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:47.104301 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:47.104301 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:47.104301 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:47.104301 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:47.104301 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:47.104301 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:47.104301 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:47.105999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.060 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:47.105999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:47.105999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:47.105999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:47.105999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:47.105999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:47.105999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:47.105999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:47.105999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:47.105999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:47.105999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:47.105999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:47.107070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:47.107070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:47.107070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:47.107070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:47.107070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:47.107070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:47.107070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:47.107070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:47.107070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:47.107070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:47.107070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:47.107070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:47.107070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:47.107346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:47.107346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:47.107596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:47.107596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:47.107596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:47.107596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:47.107596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:47.107596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:47.107596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:47.107596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:47.107596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:47.107596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:47.107596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:47.107596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:47.107596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:47.107596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:47.107596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:47.107596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:47.107995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:47.108480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:47.108480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:47.108480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:51.902930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:51.903755 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:51.905641 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:51.906680 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:51.906881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:51.906988 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:51.907100 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:51.907213 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:51.907332 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:51.907457 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:51.907582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:51.907688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.908143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:51.908143 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:51.908143 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:51.908143 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:51.908143 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:51.908143 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:51.908143 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.908143 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:51.909004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.909004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:51.909004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:51.909004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:51.909004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:51.909004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:51.909004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:51.909004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:51.909004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:51.909004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:51.909004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:51.909004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:51.909240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:51.909240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:51.909240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:51.909240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:51.909240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:51.909240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:51.909240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:51.909240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:51.909240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:51.909240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:51.909240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:51.909240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:51.909240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:51.909474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.909474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:51.910166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.910166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:51.910166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:51.910166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:51.910166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:51.910166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:51.910166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:51.910166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:51.910166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:51.910166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:51.910166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:51.910166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:51.910166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:51.910166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:51.910166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:51.910166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:51.910474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:51.910803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:51.910803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.910803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:51.974565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:51.974565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:51.974565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:51.974565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:41:51.974565 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:51.974565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:51.975304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:51.975844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:51.975844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:51.975844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:51.975844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:51.975844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:51.975844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:51.975844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:51.975844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:51.975844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:51.975844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:51.975844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:51.975844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:51.975844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.975844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:51.975844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.976262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:51.976262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:51.976262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:51.976262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:51.976262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:51.976262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:51.976262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:41:51.976262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:51.976262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:51.976262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:51.976262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:51.976262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:51.976262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:51.976682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:51.976682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:51.976682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:51.976682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:51.976682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:51.976682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:51.976682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:51.976682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:51.976682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:51.976682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:51.976682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:51.976682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:51.976682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:51.976682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:51.977040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.977040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:51.977040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.977040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:51.977040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:51.977040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:51.977040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:51.977040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:51.977040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:51.977040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:51.977040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:51.977040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:51.977040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:51.977040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:51.977040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:51.977456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:51.978012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:51.978012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:51.978012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:51.978012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:51.978012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:51.978012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:51.978012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:51.978012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.978012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:51.985610 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:51.985997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:51.986131 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:51.986262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:51.986882 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:51.987165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.987814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:51.987814 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:51.987814 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:51.987814 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:51.987814 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:51.987814 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:51.987814 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.987814 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:51.991323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.991323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:51.991323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:51.991323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:51.991323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:51.991323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:51.991323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:51.991323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:51.991323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:51.991323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:51.991323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:51.991323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:51.991323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:51.991323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:51.991323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:51.992009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:51.992009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:51.992009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:51.992009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:51.992009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:51.992009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:51.992009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:51.992009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.992009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:51.992009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.992009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:51.992009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:51.992009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:51.992009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:51.992599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:51.992599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:51.992599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:51.992599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:51.992599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:51.992599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:51.992599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:51.992599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:51.992599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:51.992599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:51.992599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:51.992599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:51.992599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:51.992971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:51.992971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:51.992971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:51.992971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:51.992971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:51.992971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:51.992971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:51.992971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:51.992971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:51.992971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.992971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:51.992971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:51.993295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:51.993905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:52.020741 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:52.021054 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:52.021178 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:52.021390 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:52.023958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:52.024147 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:52.024346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:52.024854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:52.024854 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:52.024854 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:52.024854 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:52.024854 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:52.024854 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:52.024854 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:52.024854 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:52.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:52.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:52.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:52.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:52.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:52.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:52.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:52.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:52.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:52.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:52.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:52.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:52.026180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:52.026180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:52.026180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:52.026180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:52.026180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:52.026180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:52.026180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:52.026180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:52.026180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:52.026180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:52.026180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:52.026180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:52.026180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:52.026398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:52.026398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:52.029557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:52.029557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:52.029557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:52.029557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:52.029557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:52.029557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:52.029557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:52.029557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:52.029557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:52.029557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:52.029557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:52.029557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:52.029557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:52.029557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:52.029557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:52.029557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:52.029895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:52.030210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:52.030210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:52.030210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:52.043751 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:52.044054 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:52.044175 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:52.044980 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:52.103650 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:52.103840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:52.104648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:52.104648 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:52.104648 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:52.104648 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:52.104648 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:52.104648 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:52.104648 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:52.104648 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:52.105779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.060 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:52.105779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:52.105779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:52.105779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:52.105779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:52.105779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:52.105779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:52.105779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:52.105779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:52.105779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:52.105779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:52.105779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:52.106822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:52.106822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:52.106822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:52.106822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:52.106822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:52.106822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:52.106822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:52.106822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:52.106822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:52.106822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:52.106822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:52.106822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:52.106822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:52.107413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:52.107413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:52.107914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:52.107914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:52.107914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:52.107914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:52.107914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:52.107914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:52.107914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:52.107914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:52.107914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:52.107914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:52.107914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:52.107914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:52.107914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:52.107914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:52.107914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:52.107914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:52.108386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:52.108920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:52.108920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:52.108920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:56.903786 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:56.904462 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:56.904462 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:56.907078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:56.907333 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:56.907520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:56.907709 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:56.907876 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:56.908040 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:56.908185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:56.908328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:56.908486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.909259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:56.909259 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:56.909259 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:56.909259 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:56.909259 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:56.909259 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:56.909259 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.909259 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:56.910568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.910568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:56.910568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:56.910568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:56.910568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:56.910568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:56.910568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:56.910568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:56.910568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:56.910568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:56.910568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:56.910568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:56.910964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:56.910964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:56.910964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:56.910964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:56.910964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:56.910964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:56.910964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:56.910964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:56.910964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:56.910964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:56.910964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:56.910964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:56.910964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:56.911286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.911286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:56.911903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.911903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:56.911903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:56.911903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:56.911903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:56.911903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:56.911903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:56.911903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:56.911903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:56.911903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:56.911903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:56.911903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:56.911903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:56.911903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:56.911903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:56.911903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:56.912218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:56.912651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:56.912651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.912651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:56.973097 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:56.973739 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:56.974561 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:56.974892 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:41:56.975618 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:56.975832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.976626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:56.976626 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:56.976626 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:56.976626 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:56.976626 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:56.976626 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:56.976626 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.976626 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:56.979141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.979141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:56.979141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:56.979141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:56.979141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:56.979141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:56.979141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:56.979141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:56.979141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:56.979141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:56.979141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:56.979141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:56.979141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:56.979141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:56.979141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:56.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:56.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:56.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:56.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:56.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:56.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:56.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:56.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.980121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:56.981197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.981197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:56.981197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:56.981197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:56.981197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:56.981197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:56.981197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:56.981197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:41:56.981197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:56.981197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:56.981197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:56.981197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:56.981486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:56.981486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:56.981486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:56.981486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:56.981486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:56.981486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:56.981486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:56.981486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:56.981486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:56.981486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:56.981486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:56.981486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:56.981486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:56.981486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:56.981945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:56.981945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:56.981945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.981945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:56.986565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.986565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:56.986565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:56.986565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:56.986565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:56.986565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:56.986565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:56.986565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:56.986565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:56.986565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:56.986565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:56.986565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:56.986565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:56.986565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:56.986565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:56.986565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:56.989818 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:56.990353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:56.990353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:56.990353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:56.990353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:56.990353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:56.990353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.990353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:56.990353 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:56.990353 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:56.990353 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:56.990568 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:56.990568 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:56.990568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.991110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:56.991110 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:56.991110 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:56.991110 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:56.991110 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:56.991110 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:56.991110 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.991110 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:56.992875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.992875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:41:56.992875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:56.992875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:56.992875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:56.992875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:56.992875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:56.992875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:41:56.992875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:56.992875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:41:56.992875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:56.992875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:41:56.992875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:56.992875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:41:56.992875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:41:56.993368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:56.993368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:41:56.993368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:56.993368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:41:56.993368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:41:56.993368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:56.993368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:41:56.993368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.993368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:41:56.995786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.995786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:56.995786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:56.995786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:56.995786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:56.995786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:56.995786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:56.995786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:56.995786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:56.995786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:56.995786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:56.996113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:56.996113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:56.996113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:56.996113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:56.996113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:56.996113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:56.996113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:56.996113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:56.996113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:56.996113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:56.996113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:56.996113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:56.996113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:56.996113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:56.996527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:56.996527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:56.996527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.996527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:56.996527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.996527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:56.996527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:56.996527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:56.996527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:56.996527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:56.996527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:56.996527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:56.996527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:56.996527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:56.996912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:56.997507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:56.997507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:56.997507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:56.997507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:56.997507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:56.997507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:56.997507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:56.997507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:56.997507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:56.997507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:56.997507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:57.021179 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:57.021610 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:57.021715 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:57.021877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:57.023976 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:57.024180 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:57.024377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:57.024967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:57.024967 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:57.024967 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:57.024967 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:57.024967 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:57.024967 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:57.024967 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:57.024967 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:57.025672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:57.025672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:57.025672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:57.025672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:57.025672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:57.025672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:57.025672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:57.025672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:57.025672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:57.025672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:57.025672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:57.025672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:57.026076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:57.026076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:57.026076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:57.026076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:57.026076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:57.026076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:57.026076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:57.026076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:57.026076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:57.026076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:57.026076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:57.026076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:57.026076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:57.026365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:57.026365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:57.026502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:57.026502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:57.026502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:57.026502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:57.026502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:57.026502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:57.026502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:57.026502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:57.026502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:57.026502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:57.026502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:57.026502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:57.026502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:57.026502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:57.026502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:57.026502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:57.026833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:57.027277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:57.027277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:57.027277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:41:57.044179 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:41:57.044469 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:41:57.044590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:41:57.045187 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:41:57.085030 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:41:57.085237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:57.087602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:41:57.087602 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:57.087602 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:41:57.087602 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:41:57.087602 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:57.087602 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:41:57.087602 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:57.087602 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:41:57.088352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:57.088352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:41:57.088352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:57.088352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:41:57.088352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:57.088352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:57.088352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:57.088352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:57.088352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:57.088352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:57.088352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:41:57.088352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:57.088825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:41:57.088825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:57.088825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:41:57.088825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:57.088825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:41:57.088825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:41:57.088825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:57.088825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:41:57.088825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:57.088825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:41:57.088825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:41:57.088825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:57.088825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:41:57.089235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:57.089235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:41:57.089235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:57.089235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:41:57.089235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:41:57.089235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:41:57.089235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:41:57.089235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:41:57.089235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:57.089235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:41:57.089235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:41:57.089235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:57.089235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:41:57.089235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:41:57.089235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:41:57.090736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:41:57.091526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:41:57.091526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:41:57.091526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:41:57.091526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:41:57.091526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:41:57.091526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:01.904186 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:01.905582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:01.905582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:01.905582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:01.905582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:01.905582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:01.905812 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:01.905812 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:01.905891 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:01.905927 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:01.906016 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:01.906118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.907872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:01.907872 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:01.907872 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:01.907872 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:01.907872 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:01.907872 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:01.907872 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.907872 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:01.907872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.907872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:01.907872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:01.907872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:01.907872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:01.907872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:01.907872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:01.908180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:01.908180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:01.908180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:01.908180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:01.908180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:01.908180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:01.908180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:01.908180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:01.908180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:01.908180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:01.908180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:01.908180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:01.908180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:01.908180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:01.908416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:01.908416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:01.908416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:01.908416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:01.908416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.908416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:01.908416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.908416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:01.908416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:01.908416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:01.908416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:01.908416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:01.908416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:01.908416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:01.908710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:01.909079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:01.909079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:01.909079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:01.909079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:01.909079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:01.909079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:01.909079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:01.909079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:01.909079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.909079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:01.975453 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:01.975934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:01.975934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:01.975934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:42:01.976470 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:01.976660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.977046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:01.977046 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:01.977046 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:01.977046 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:01.977046 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:01.977046 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:01.977046 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.977046 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:01.977862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.977862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:01.977862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:01.977862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:01.977862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:01.977862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:01.977862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:01.977862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:01.977862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:01.977862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:01.977862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:01.977862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:01.977862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:01.977862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:01.977862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:01.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:01.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:01.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:01.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:01.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:01.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:01.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:01.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.978181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:01.981957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.981957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:01.981957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:01.981957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:01.981957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:01.981957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:01.981957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:01.981957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:42:01.981957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:01.981957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:01.981957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:01.981957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:01.982239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:01.982239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:01.982239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:01.982239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:01.982239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:01.982239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:01.982239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:01.982239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:01.982239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:01.982239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:01.982239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:01.982239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:01.982239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:01.982239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:01.982527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:01.982527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:01.982527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.982527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:01.982527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.982527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:01.982527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:01.982527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:01.982527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:01.982527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:01.982527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:01.982527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:01.982527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:01.982527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:01.982795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:01.983117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:01.983117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:01.983117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:01.983117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:01.983117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:01.983117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:01.983117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:01.983117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:01.983117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:01.983117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:01.983117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.983117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:01.987576 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:01.988165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:01.988165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:01.988165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:01.988372 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:01.988582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:01.991061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:01.991430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:01.991430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:01.991430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:01.991430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:01.991430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:01.991430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:01.991430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:01.991430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:01.991430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:01.991430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:01.991430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:01.991430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:01.991430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.991430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:01.991430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.991743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:01.991743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:01.991743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:01.991743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:01.991743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:01.991743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:01.991743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:01.991743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:01.991743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:01.991743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:01.991743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:01.991743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:01.991743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:01.991952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:01.991952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:01.991952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:01.991952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:01.991952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:01.991952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:01.991952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:01.991952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:01.991952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:01.991952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:01.991952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:01.991952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:01.991952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:01.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:01.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:01.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:01.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:01.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:01.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:01.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:01.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:01.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:01.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:01.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:01.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:01.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:01.992566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:01.992906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:01.992906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:01.992906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:01.992906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:01.992906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:01.992906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:01.992906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:01.992906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:02.021586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:02.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:02.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:02.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:02.022407 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:02.022640 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:02.022899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:02.024710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:02.024710 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:02.024710 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:02.024710 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:02.024710 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:02.024710 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:02.024710 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:02.024710 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:02.024710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:02.024710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:02.024710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:02.024710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:02.024710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:02.024710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:02.024710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:02.025102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:02.025102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:02.025102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:02.025102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:02.025102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:02.025102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:02.025102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:02.025102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:02.025102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:02.025102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:02.025102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:02.025102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:02.025102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:02.025102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:02.025395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:02.025395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:02.025395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:02.025395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:02.025395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:02.025395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:02.025395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:02.025395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:02.025395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:02.025395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:02.025395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:02.025395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:02.025395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:02.025395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:02.025685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:02.026065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:02.026065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:02.026065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:02.026065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:02.026065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:02.026065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:02.026065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:02.026065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:02.026065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:02.026065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:02.044555 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:02.045407 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:02.045407 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:02.045942 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:02.079938 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:02.080108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:02.080582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:02.080582 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:02.080582 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:02.080582 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:02.080582 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:02.080582 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:02.080582 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:02.080582 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:02.082739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:02.082739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:02.082739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:02.082739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:02.082739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:02.082739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:02.082739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:02.082739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:02.082739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:02.082739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:02.082739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:02.082739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:02.083035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:02.083035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:02.083035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:02.083035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:02.083035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:02.083035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:02.083035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:02.083035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:02.083035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:02.083035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:02.083035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:02.083035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:02.083035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:02.083251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:02.083251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:02.083251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:02.083251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:02.083251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:02.083251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:02.083251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:02.083251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:02.083251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:02.083251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:02.083251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:02.083251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:02.083251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:02.083251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:02.083251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:02.083498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:02.083861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:02.083861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:02.083861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:02.083861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:02.083861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:02.083861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:06.904708 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:06.905145 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:06.905145 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:06.906897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:06.906897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:06.906897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:06.906897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:06.906897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:06.907082 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:06.907082 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:06.907082 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:06.907082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.907082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:06.907082 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:06.907082 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:06.907082 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:06.907082 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:06.907082 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:06.907082 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.907082 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:06.907668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.907668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:06.907668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:06.907668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:06.907668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:06.907668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:06.907668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:06.907668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:06.907668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:06.907668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:06.907668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:06.907668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:06.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:06.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:06.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:06.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:06.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:06.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:06.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:06.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:06.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:06.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:06.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:06.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:06.907927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:06.908273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.908273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:06.908474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.908474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:06.908474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:06.908474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:06.908474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:06.908474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:06.908474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:06.908474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:06.908474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:06.908474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:06.908474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:06.908474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:06.908474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:06.908474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:06.908474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:06.908474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:06.908812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:06.909321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:06.909321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.909321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:06.975875 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:06.976113 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:06.976207 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:06.976308 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:42:06.977270 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:06.977270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.977399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:06.977399 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:06.977399 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:06.977399 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:06.977399 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:06.977399 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:06.977399 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.977399 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:06.977920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.977920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:06.977920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:06.977920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:06.977920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:06.977920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:06.977920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:06.977920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:06.977920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:06.977920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:06.977920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:06.977920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:06.977920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:06.977920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:06.977920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:06.978190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:06.978190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:06.978190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:06.978190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:06.978190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:06.978190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:06.978190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:06.978190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.978190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:06.978479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.978479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:06.978479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:06.978479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:06.978479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:06.978479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:06.978479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:06.978479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:42:06.978479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:06.978479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:06.978479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:06.978479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:06.978735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:06.978735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:06.978735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:06.978735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:06.978735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:06.978735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:06.978735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:06.978735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:06.978735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:06.978735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:06.978735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:06.978735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:06.978735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:06.978735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:06.979081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:06.979081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:06.979081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.979081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:06.979200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.979200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:06.979200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:06.979200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:06.979200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:06.979200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:06.979200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:06.979200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:06.979200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:06.979200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:06.979200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:06.979200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:06.979200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:06.979200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:06.979200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:06.979200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:06.979493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:06.979807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:06.979807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:06.979807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:06.979807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:06.979807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:06.979807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.979807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:06.987883 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:06.988067 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:06.988186 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:06.988321 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:06.989007 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:06.989007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:06.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:06.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:06.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:06.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:06.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:06.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.989138 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:06.989817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.989817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:06.989817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:06.989817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:06.989817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:06.989817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:06.989817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:06.989817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:06.989817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:06.989817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:06.989817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:06.989817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:06.989817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:06.989817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:06.989817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:06.990241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:06.990241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:06.990241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:06.990241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:06.990241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:06.990241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:06.990241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:06.990241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.990241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:06.990749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.990749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:06.990749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:06.990749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:06.990749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:06.990749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:06.990749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:06.990749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:06.990749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:06.990749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:06.990749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:06.991076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:06.991076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:06.991076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:06.991076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:06.991076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:06.991076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:06.991076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:06.991076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:06.991076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:06.991076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:06.991076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:06.991076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:06.991076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:06.991076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:06.991499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:06.991499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:06.991499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.991499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:06.991872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:06.992242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:06.992629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:06.992629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:06.992629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:06.992629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:07.022054 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:07.022349 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:07.022452 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:07.022652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:07.023673 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:07.023673 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:07.023673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:07.023893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:07.023893 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:07.023893 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:07.023893 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:07.023893 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:07.023893 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:07.023893 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:07.023893 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:07.024630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:07.024630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:07.024630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:07.024630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:07.024630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:07.024630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:07.024630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:07.024630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:07.024630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:07.024630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:07.024630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:07.024630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:07.024934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:07.024934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:07.024934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:07.024934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:07.024934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:07.024934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:07.024934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:07.024934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:07.024934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:07.024934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:07.024934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:07.024934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:07.024934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:07.025201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:07.025201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:07.025368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:07.025368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:07.025368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:07.025368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:07.025368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:07.025368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:07.025368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:07.025368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:07.025368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:07.025368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:07.025368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:07.025368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:07.025368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:07.025368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:07.025368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:07.025368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:07.026048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:07.026482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:07.026482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:07.026482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:07.044961 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:07.045172 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:07.045291 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:07.045923 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:07.072617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:07.072617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:07.072814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:07.072814 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:07.072814 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:07.072814 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:07.072814 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:07.072814 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:07.072814 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:07.072814 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:07.073368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:07.073368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:07.073368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:07.073368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:07.073368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:07.073368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:07.073368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:07.073368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:07.073368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:07.073368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:07.073368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:07.073368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:07.073664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:07.073664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:07.073664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:07.073664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:07.073664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:07.073664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:07.073664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:07.073664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:07.073664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:07.073664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:07.073664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:07.073664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:07.073664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:07.073973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:07.073973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:07.074123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:07.074123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:07.074123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:07.074123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:07.074123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:07.074123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:07.074123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:07.074123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:07.074123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:07.074123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:07.074123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:07.074123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:07.074123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:07.074123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:07.074123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:07.074123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:07.074470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:07.074957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:07.074957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:07.074957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:11.905033 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:11.905451 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:11.905508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:11.906868 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:11.907059 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:11.907187 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:11.907251 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:11.907369 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:11.907501 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:11.907634 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:11.907743 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:11.907970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.908427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:11.908427 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:11.908427 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:11.908427 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:11.908427 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:11.908427 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:11.908427 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.908427 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:11.909185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.909185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:11.909185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:11.909185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:11.909185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:11.909185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:11.909185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:11.909185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:11.909185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:11.909185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:11.909185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:11.909185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:11.909477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:11.909477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:11.909477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:11.909477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:11.909477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:11.909477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:11.909477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:11.909477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:11.909477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:11.909477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:11.909477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:11.909477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:11.909477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:11.909828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.909828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:11.910141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.910141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:11.910141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:11.910141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:11.910141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:11.910141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:11.910141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:11.910141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:11.910141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:11.910141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:11.910141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:11.910141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:11.910141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:11.910141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:11.910141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:11.910141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:11.910686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:11.911194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:11.911194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.911194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:11.976249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:11.978020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:11.978183 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:11.978308 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:42:11.978841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:11.978987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.979623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:11.979623 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:11.979623 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:11.979623 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:11.979623 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:11.979623 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:11.979623 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.979623 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:11.980403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.980403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:11.980403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:11.980403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:11.980403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:11.980403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:11.980403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:11.980403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:11.980403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:11.980403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:11.980403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:11.980403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:11.980403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:11.980403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:11.980403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:11.980819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:11.980819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:11.980819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:11.980819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:11.980819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:11.980819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:11.980819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:11.980819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.980819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:11.981325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.981325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:11.981325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:11.981325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:11.981325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:11.981325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:11.981325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:11.981325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:42:11.981325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:11.981325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:11.981325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:11.981325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:11.981695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:11.981695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:11.981695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:11.981695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:11.981695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:11.981695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:11.981695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:11.981695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:11.981695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:11.981695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:11.981695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:11.981695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:11.981695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:11.981695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:11.981971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:11.981971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:11.981971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.981971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:11.982462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.982462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:11.982462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:11.982462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:11.982462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:11.982462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:11.982462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:11.982462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:11.982462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:11.982462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:11.982462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:11.982462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:11.982462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:11.982462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:11.982462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:11.982462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:11.982807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:11.983199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:11.983199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:11.983199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:11.983199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:11.983199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:11.983199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.983199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:11.988232 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:11.991961 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:11.992092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:11.992187 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:11.992637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:11.992908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:11.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:11.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:11.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:11.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:11.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:11.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:11.995663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.995663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:11.995663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:11.995663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:11.995663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:11.995663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:11.995663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:11.995663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:11.995663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:11.995663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:11.995663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:11.995663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:11.995663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:11.995663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:11.995663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:11.995966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:11.995966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:11.995966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:11.995966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:11.995966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:11.995966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:11.995966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:11.995966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:11.995966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:12.000162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:12.000162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:12.000162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:12.000162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:12.000162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:12.000162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:12.000162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:12.000162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:12.000162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:12.000162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:12.000162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:12.000456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:12.000456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:12.000456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:12.000456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:12.000456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:12.000456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:12.000456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:12.000456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:12.000456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:12.000456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:12.000456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:12.000456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:12.000456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:12.000456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:12.000792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:12.000792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:12.000792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:12.000792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:12.001446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:12.001773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:12.002124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:12.002124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:12.002124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:12.002124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:12.022582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:12.023035 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:12.023168 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:12.023371 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:12.023775 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:12.024036 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:12.024308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:12.024989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:12.024989 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:12.024989 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:12.024989 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:12.024989 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:12.024989 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:12.024989 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:12.024989 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:12.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:12.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:12.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:12.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:12.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:12.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:12.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:12.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:12.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:12.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:12.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:12.025901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:12.026265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:12.026265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:12.026265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:12.026265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:12.026265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:12.026265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:12.026265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:12.026265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:12.026265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:12.026265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:12.026265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:12.026265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:12.026265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:12.026626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:12.026626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:12.027005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:12.027005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:12.027005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:12.027005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:12.027005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:12.027005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:12.027005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:12.027005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:12.027005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:12.027005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:12.027005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:12.027005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:12.027005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:12.027005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:12.027005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:12.027005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:12.027425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:12.027941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:12.027941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:12.027941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:12.045697 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:12.048012 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:12.048104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:12.048918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:12.087656 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:12.087876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:12.088214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:12.088214 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:12.088214 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:12.088214 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:12.088214 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:12.088214 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:12.088214 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:12.088214 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:12.088884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:12.088884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:12.088884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:12.088884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:12.088884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:12.088884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:12.088884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:12.088884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:12.088884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:12.088884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:12.088884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:12.088884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:12.089150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:12.089150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:12.089150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:12.089150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:12.089150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:12.089150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:12.089150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:12.089150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:12.089150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:12.089150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:12.089150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:12.089150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:12.089150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:12.089418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:12.089418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:12.089911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:12.089911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:12.089911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:12.089911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:12.089911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:12.089911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:12.089911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:12.089911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:12.089911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:12.089911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:12.089911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:12.089911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:12.089911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:12.089911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:12.089911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:12.089911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:12.090244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:12.090639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:12.090639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:12.090639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:16.905485 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:16.905971 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:16.905971 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:16.906768 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:16.906908 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:16.907018 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:16.907128 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:16.907209 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:16.907302 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:16.907404 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:16.907513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:16.907630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.908062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:16.908062 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:16.908062 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:16.908062 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:16.908062 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:16.908062 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:16.908062 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.908062 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:16.908772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.908772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:16.908772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:16.908772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:16.908772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:16.908772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:16.908772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:16.908772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:16.908772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:16.908772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:16.908772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:16.908772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:16.909216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:16.909216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:16.909216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:16.909216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:16.909216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:16.909216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:16.909216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:16.909216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:16.909216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:16.909216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:16.909216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:16.909216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:16.909216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:16.909505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.909505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:16.909505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.909505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:16.909505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:16.909505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:16.909505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:16.909505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:16.909505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:16.909505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:16.909505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:16.909505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:16.909505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:16.909505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:16.909505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:16.909937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:16.910496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:16.910496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:16.910496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:16.910496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:16.910496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.910496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:16.978609 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:16.978609 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:16.978609 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:16.978609 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:42:16.978893 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:16.979008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.979381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:16.979381 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:16.979381 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:16.979381 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:16.979381 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:16.979381 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:16.979381 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.979381 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:16.979910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.979910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:16.979910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:16.979910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:16.979910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:16.979910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:16.979910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:16.979910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:16.979910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:16.979910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:16.979910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:16.979910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:16.979910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:16.979910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:16.979910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:16.980139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:16.980139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:16.980139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:16.980139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:16.980139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:16.980139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:16.980139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:16.980139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.980139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:16.980790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.980790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:16.980790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:16.980790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:16.980790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:16.980790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:16.980790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:16.980790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:42:16.980790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:16.980790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:16.980790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:16.980790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:16.981046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:16.981046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:16.981046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:16.981046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:16.981046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:16.981046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:16.981046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:16.981046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:16.981046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:16.981046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:16.981046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:16.981046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:16.981046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:16.981046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:16.981325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:16.981325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:16.981325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.981325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:16.981791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.981791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:16.981791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:16.981791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:16.981791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:16.981791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:16.981791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:16.981791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:16.981791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:16.981791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:16.981791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:16.981791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:16.981791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:16.981791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:16.981791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:16.981791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:16.982191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:16.982548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:16.982548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:16.982548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:16.982548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:16.982548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:16.982548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.982548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:16.991856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:16.992227 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:16.992380 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:16.992589 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:16.992930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:16.993108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.993548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:16.993548 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:16.993548 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:16.993548 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:16.993548 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:16.993548 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:16.993548 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.993548 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:16.994459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.994459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:16.994459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:16.994459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:16.994459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:16.994459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:16.994459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:16.994459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:16.994459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:16.994459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:16.994459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:16.994459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:16.994459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:16.994459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:16.994459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:16.994887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:16.994887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:16.994887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:16.994887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:16.994887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:16.994887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:16.994887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:16.994887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.994887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:16.995531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.995531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:16.995531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:16.995531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:16.995531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:16.995531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:16.995531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:16.995531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:16.995531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:16.995531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:16.995531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:16.995779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:16.995779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:16.995779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:16.995779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:16.995779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:16.995779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:16.995779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:16.995779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:16.995779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:16.995779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:16.995779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:16.995779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:16.995779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:16.995779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:16.996030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:16.996030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:16.996030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.996030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:16.996854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:16.997168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:16.997666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:16.997666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:16.997666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:16.997666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:17.023001 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:17.023479 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:17.023657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:17.023817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:17.024231 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:17.024430 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:17.024740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:17.025274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:17.025274 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:17.025274 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:17.025274 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:17.025274 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:17.025274 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:17.025274 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:17.025274 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:17.026001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:17.026001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:17.026001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:17.026001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:17.026001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:17.026001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:17.026001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:17.026001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:17.026001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:17.026001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:17.026001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:17.026001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:17.026253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:17.026253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:17.026253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:17.026253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:17.026253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:17.026253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:17.026253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:17.026253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:17.026253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:17.026253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:17.026253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:17.026253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:17.026253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:17.026516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:17.026516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:17.026747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:17.026747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:17.026747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:17.026747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:17.026747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:17.026747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:17.026747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:17.026747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:17.026747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:17.026747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:17.026747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:17.026747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:17.026747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:17.026747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:17.026747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:17.026747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:17.027039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:17.027395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:17.027395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:17.027395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:17.048069 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:17.048487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:17.048722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:17.049481 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:17.085334 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:17.085591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:17.086177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:17.086177 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:17.086177 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:17.086177 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:17.086177 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:17.086177 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:17.086177 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:17.086177 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:17.087054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:17.087054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:17.087054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:17.087054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:17.087054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:17.087054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:17.087054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:17.087054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:17.087054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:17.087054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:17.087054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:17.087054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:17.087497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:17.087497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:17.087497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:17.087497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:17.087497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:17.087497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:17.087497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:17.087497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:17.087497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:17.087497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:17.087497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:17.087497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:17.087497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:17.087991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:17.087991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:17.088193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:17.088193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:17.088193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:17.088193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:17.088193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:17.088193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:17.088193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:17.088193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:17.088193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:17.088193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:17.088193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:17.088193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:17.088193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:17.088193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:17.088193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:17.088193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:17.088719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:17.089302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:17.089302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:17.089302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:21.906940 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:21.908149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:21.908271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:21.909407 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:21.909586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:21.909707 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:21.909811 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:21.909909 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:21.910006 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:21.910103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:21.910199 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:21.910298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:21.910733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:21.910733 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:21.910733 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:21.910733 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:21.910733 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:21.910733 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:21.910733 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:21.910733 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:21.911348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:21.911348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:21.911348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:21.911348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:21.911348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:21.911348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:21.911348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:21.911348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:21.911348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:21.911348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:21.911348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:21.911348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:21.911625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:21.911625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:21.911625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:21.911625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:21.911625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:21.911625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:21.911625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:21.911625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:21.911625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:21.911625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:21.911625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:21.911625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:21.911625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:21.911849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:21.911849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:21.912061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:21.912061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:21.912061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:21.912061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:21.912061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:21.912061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:21.912061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:21.912061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:21.912061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:21.912061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:21.912061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:21.912061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:21.912061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:21.912061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:21.912061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:21.912061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:21.912326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:21.912704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:21.912704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:21.912704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:21.979068 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:21.979510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:21.979649 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:21.979754 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:42:21.980077 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:21.980231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:21.980712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:21.980712 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:21.980712 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:21.980712 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:21.980712 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:21.980712 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:21.980712 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:21.980712 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:21.981340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:21.981340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:21.981340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:21.981340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:21.981340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:21.981340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:21.981340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:21.981340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:21.981340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:21.981340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:21.981340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:21.981340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:21.981340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:21.981340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:21.981340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:21.981898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:21.981898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:21.981898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:21.981898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:21.981898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:21.981898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:21.981898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:21.981898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:21.981898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:21.982176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:21.982176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:21.982176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:21.982176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:21.982176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:21.982176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:21.982176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:21.982176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:42:21.982176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:21.982176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:21.982176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:21.982176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:21.982528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:21.982528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:21.982528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:21.982528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:21.982528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:21.982528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:21.982528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:21.982528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:21.982528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:21.982528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:21.982528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:21.982528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:21.982528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:21.982528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:21.982924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:21.982924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:21.982924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:21.982924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:21.982924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:21.982924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:21.982924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:21.982924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:21.982924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:21.982924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:21.982924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:21.982924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:21.982924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:21.982924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:21.983320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:21.983884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:21.983884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:21.983884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:21.983884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:21.983884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:21.983884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:21.983884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:21.983884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:21.983884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:21.983884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:21.983884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:21.983884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:21.998392 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:21.998900 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:21.998999 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:21.999163 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:21.999581 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:21.999861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.000521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:22.000521 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:22.000521 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:22.000521 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:22.000521 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:22.000521 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:22.000521 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.000521 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:22.001324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.001324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:22.001324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:22.001324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:22.001324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:22.001324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:22.001324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:22.001324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:22.001324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:22.001324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:22.001324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:22.001324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:22.001324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:22.001324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:22.001324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:22.001627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:22.001627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:22.001627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:22.001627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:22.001627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:22.001627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:22.001627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:22.001627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.001627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:22.002258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.002258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:22.002258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:22.002258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:22.002258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:22.002258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:22.002258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:22.002258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:22.002258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:22.002258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:22.002258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:22.002525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:22.002525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:22.002525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:22.002525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:22.002525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:22.002525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:22.002525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:22.002525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:22.002525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:22.002525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:22.002525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:22.002525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:22.002525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:22.002525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:22.002789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:22.002789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:22.002789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.002789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:22.003570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:22.003896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:22.004228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:22.004228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:22.004228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.004228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:22.023600 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:22.023969 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:22.024082 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:22.024243 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:22.024536 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:22.024757 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:22.024994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.025412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:22.025412 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:22.025412 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:22.025412 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:22.025412 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:22.025412 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:22.025412 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.025412 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:22.026268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.026268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:22.026268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:22.026268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:22.026268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:22.026268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:22.026268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:22.026268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:22.026268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:22.026268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:22.026268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:22.026268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:22.026512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:22.026512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:22.026512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:22.026512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:22.026512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:22.026512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:22.026512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:22.026512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:22.026512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:22.026512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:22.026512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:22.026512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:22.026512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:22.026847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.026847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:22.027354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.027354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:22.027354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:22.027354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:22.027354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:22.027354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:22.027354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:22.027354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:22.027354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:22.027354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:22.027354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:22.027354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:22.027354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:22.027354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:22.027354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:22.027354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:22.027693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:22.028029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:22.028029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.028029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:22.048518 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:22.048939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:22.049083 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:22.049997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:22.060185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=242 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:42:22.096183 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:22.096550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.097194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:22.097194 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:22.097194 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:22.097194 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:22.097194 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:22.097194 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:22.097194 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.097194 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:22.097922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.048 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.097922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:22.097922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:22.097922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:22.097922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:22.097922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:22.097922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:22.097922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:22.097922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:22.097922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:22.097922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:22.097922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:22.098183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:22.098183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:22.098183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:22.098183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:22.098183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:22.098183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:22.098183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:22.098183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:22.098183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:22.098183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:22.098183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:22.098183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:22.098183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:22.098399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.098399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:22.098969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.098969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:22.098969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:22.098969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:22.098969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:22.098969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:22.098969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:22.098969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:22.098969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:22.098969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:22.098969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:22.098969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:22.098969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:22.098969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:22.098969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:22.098969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:22.099257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:22.099578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:22.099578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:22.099578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:26.908614 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:26.909086 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:26.909086 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:26.912667 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:26.912667 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:26.912667 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:26.912667 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:26.912667 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:26.912771 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:26.912771 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:26.912771 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:26.912771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:26.912771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:26.912771 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:26.912771 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:26.912771 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:26.912771 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:26.912771 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:26.912771 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:26.912771 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:26.912983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:26.912983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:26.912983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:26.912983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:26.912983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:26.912983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:26.912983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:26.912983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:26.912983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:26.912983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:26.912983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:26.912983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:26.913170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:26.913170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:26.913170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:26.913170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:26.913170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:26.913170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:26.913170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:26.913170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:26.913170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:26.913170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:26.913170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:26.913170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:26.913170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:26.913369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:26.913369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:26.913369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:26.913369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:26.913369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:26.913369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:26.913369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:26.913369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:26.913369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:26.913369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:26.913369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:26.913369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:26.913369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:26.913369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:26.913369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:26.913633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:26.913919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:26.913919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:26.913919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:26.913919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:26.913919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:26.913919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:26.979616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:26.979890 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:26.980006 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:26.980091 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:42:26.980413 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:26.980605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:26.981243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:26.981243 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:26.981243 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:26.981243 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:26.981243 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:26.981243 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:26.981243 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:26.981243 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:26.981805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:26.981805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:26.981805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:26.981805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:26.981805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:26.981805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:26.981805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:26.981805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:26.981805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:26.981805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:26.981805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:26.981805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:26.981805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:26.981805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:26.981805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:26.982866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:26.982866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:26.982866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:26.982866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:26.982866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:26.982866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:26.982866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:26.982866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:26.982866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:26.982866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:26.982866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:26.982866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:26.982866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:26.982866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:26.983288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:26.983288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:26.983288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:42:26.983288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:26.983288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:26.983288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:26.983288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:26.983288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:26.983288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:26.983288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:26.983288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:26.983288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:26.983288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:26.983648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:26.983648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:26.983648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:26.983648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:26.983648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:26.983648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:26.983648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:26.983648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:26.983648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:26.983648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:26.983648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:26.983648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:26.986660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:26.986660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:26.986660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:26.986660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:26.986660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:26.986660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:26.986660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:26.986660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:26.986660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:26.986660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:26.986660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:26.986660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:26.986660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:26.986660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:26.986660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:26.986660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:26.987029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:26.987569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:26.987569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:26.987569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:26.987569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:26.987569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:26.987569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:26.987569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:26.998868 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:26.999317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:26.999317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:26.999317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:26.999629 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:26.999831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.000482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:27.000482 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:27.000482 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:27.000482 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:27.000482 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:27.000482 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:27.000482 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.000482 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:27.002972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.002972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:27.002972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:27.002972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:27.002972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:27.002972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:27.002972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:27.002972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:27.002972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:27.002972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:27.002972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:27.002972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:27.002972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:27.002972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:27.002972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:27.003963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:27.003963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:27.003963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:27.003963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:27.003963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:27.003963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:27.003963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:27.003963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.003963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:27.004518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.004518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:27.004518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:27.004518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:27.004518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:27.004518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:27.004518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:27.004518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:27.004518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:27.004518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:27.004518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:27.005182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:27.005182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:27.005182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:27.005182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:27.005182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:27.005182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:27.005182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:27.005182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:27.005182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:27.005182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:27.005182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:27.005182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:27.005182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:27.005182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:27.006021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:27.006021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:27.006021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.006021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:27.006021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.006021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:27.006021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:27.006021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:27.006021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:27.006021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:27.006021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:27.006021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:27.006021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:27.006021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:27.007273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:27.008026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:27.008026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:27.008026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:27.008026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:27.008026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:27.008026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:27.008026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:27.008026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:27.008026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:27.008026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.008026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:27.024078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:27.024355 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:27.024469 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:27.024671 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:27.024989 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:27.025177 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:27.025443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.025896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:27.025896 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:27.025896 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:27.025896 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:27.025896 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:27.025896 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:27.025896 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.025896 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:27.026496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.026496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:27.026496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:27.026496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:27.026496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:27.026496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:27.026496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:27.026496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:27.026496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:27.026496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:27.026496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:27.026496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:27.026793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:27.026793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:27.026793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:27.026793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:27.026793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:27.026793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:27.026793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:27.026793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:27.026793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:27.026793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:27.026793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:27.026793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:27.026793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:27.027025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.027025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:27.027318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.027318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:27.027318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:27.027318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:27.027318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:27.027318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:27.027318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:27.027318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:27.027318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:27.027318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:27.027318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:27.027318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:27.027318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:27.027318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:27.027318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:27.027318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:27.028249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:27.028588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:27.028588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.028588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:27.050605 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:27.050605 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:27.050605 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:27.050605 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:27.104678 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:27.104955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.105446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:27.105446 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:27.105446 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:27.105446 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:27.105446 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:27.105446 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:27.105446 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.105446 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:27.106204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.056 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.106204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:27.106204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:27.106204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:27.106204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:27.106204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:27.106204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:27.106204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:27.106204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:27.106204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:27.106204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:27.106204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:27.107116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:27.107116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:27.107116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:27.107116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:27.107116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:27.107116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:27.107116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:27.107116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:27.107116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:27.107116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:27.107116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:27.107116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:27.107116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:27.109084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.109084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:27.109084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.109084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:27.109084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:27.109084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:27.109084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:27.109084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:27.109084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:27.109084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:27.109084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:27.109084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:27.109084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:27.109084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:27.109084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:27.109421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:27.109793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:27.109793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:27.109793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:27.109793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:27.109793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:27.109793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:31.908773 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:31.909301 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:31.909301 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:31.910029 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:31.910192 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:31.910275 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:31.910457 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:31.910507 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:31.910745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:31.910883 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:31.911243 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:31.911243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:31.911593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:31.911593 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:31.911593 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:31.911593 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:31.911593 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:31.911593 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:31.911593 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:31.911593 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:31.913958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:31.913958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:31.913958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:31.913958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:31.913958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:31.913958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:31.913958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:31.913958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:31.913958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:31.913958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:31.913958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:31.913958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:31.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:31.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:31.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:31.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:31.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:31.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:31.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:31.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:31.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:31.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:31.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:31.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:31.914352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:31.914644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:31.914644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:31.914644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:31.914644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:31.914644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:31.914644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:31.914644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:31.914644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:31.914644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:31.914644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:31.914644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:31.914644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:31.914644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:31.914644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:31.914644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:31.914991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:31.915480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:31.915480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:31.915480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:31.915480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:31.915480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:31.915480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:31.980086 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:31.980325 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:31.980421 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:31.980578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:42:31.980880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:31.981064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:31.981481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:31.981481 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:31.981481 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:31.981481 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:31.981481 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:31.981481 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:31.981481 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:31.981481 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:31.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:31.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:31.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:31.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:31.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:31.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:31.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:31.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:31.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:31.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:31.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:31.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:31.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:31.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:31.982010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:31.982318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:31.982318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:31.982318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:31.982318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:31.982318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:31.982318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:31.982318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:31.982318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:31.982318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:31.982695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:31.982695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:31.982695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:31.982695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:31.982695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:31.982695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:31.982695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:31.982695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:42:31.982695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:31.982695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:31.982695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:31.982695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:31.982939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:31.982939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:31.982939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:31.982939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:31.982939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:31.982939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:31.982939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:31.982939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:31.982939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:31.982939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:31.982939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:31.982939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:31.982939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:31.982939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:31.983165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:31.983165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:31.983165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:31.983165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:31.983462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:31.983462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:31.983462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:31.983462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:31.983462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:31.983462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:31.983462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:31.983462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:31.983462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:31.983462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:31.983462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:31.983462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:31.983462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:31.983462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:31.983462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:31.983462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:31.983803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:31.984122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:31.984122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:31.984122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:31.984122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:31.984122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:31.984122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:31.984122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:31.999333 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:31.999666 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:31.999811 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:31.999957 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:32.000306 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:32.000627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.001174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:32.001174 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:32.001174 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:32.001174 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:32.001174 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:32.001174 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:32.001174 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.001174 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:32.002004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.002004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:32.002004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:32.002004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:32.002004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:32.002004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:32.002004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:32.002004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:32.002004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:32.002004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:32.002004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:32.002004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:32.002004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:32.002004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:32.002004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:32.002270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:32.002270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:32.002270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:32.002270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:32.002270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:32.002270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:32.002270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:32.002270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.002270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:32.003014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.003014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:32.003014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:32.003014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:32.003014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:32.003014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:32.003014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:32.003014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:32.003014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:32.003014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:32.003014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:32.003218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:32.003218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:32.003218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:32.003218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:32.003218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:32.003218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:32.003218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:32.003218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:32.003218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:32.003218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:32.003218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:32.003218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:32.003218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:32.003218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:32.003433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:32.003433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:32.003433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.003433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:32.004151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:32.004432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:32.004743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:32.004743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:32.004743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.004743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:32.024556 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:32.024832 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:32.024955 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:32.025133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:32.025434 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:32.025709 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:32.025997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.026548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:32.026548 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:32.026548 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:32.026548 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:32.026548 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:32.026548 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:32.026548 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.026548 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:32.027393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.027393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:32.027393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:32.027393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:32.027393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:32.027393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:32.027393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:32.027393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:32.027393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:32.027393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:32.027393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:32.027393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:32.027641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:32.027641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:32.027641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:32.027641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:32.027641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:32.027641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:32.027641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:32.027641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:32.027641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:32.027641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:32.027641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:32.027641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:32.027641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:32.027847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.027847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:32.028416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.028416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:32.028416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:32.028416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:32.028416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:32.028416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:32.028416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:32.028416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:32.028416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:32.028416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:32.028416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:32.028416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:32.028416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:32.028416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:32.028416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:32.028416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:32.028786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:32.029068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:32.029068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.029068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:32.049556 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:32.049860 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:32.050000 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:32.050945 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:32.086560 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:32.086728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.087110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:32.087110 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:32.087110 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:32.087110 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:32.087110 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:32.087110 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:32.087110 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.087110 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:32.087694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.087694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:32.087694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:32.087694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:32.087694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:32.087694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:32.087694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:32.087694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:32.087694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:32.087694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:32.087694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:32.087694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:32.087920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:32.087920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:32.087920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:32.087920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:32.087920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:32.087920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:32.087920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:32.087920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:32.087920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:32.087920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:32.087920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:32.087920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:32.087920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:32.088138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.088138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:32.088430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.088430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:32.088430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:32.088430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:32.088430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:32.088430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:32.088430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:32.088430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:32.088430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:32.088430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:32.088430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:32.088430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:32.088430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:32.088430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:32.088430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:32.088430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:32.088746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:32.089238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:32.089238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:32.089238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:36.909800 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:36.910376 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:36.910376 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:36.910901 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:36.911052 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:36.911181 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:36.911302 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:36.911421 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:36.911572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:36.911710 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:36.911842 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:36.911972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:36.915669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:36.915669 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:36.915669 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:36.915669 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:36.915669 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:36.915669 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:36.915669 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:36.915669 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:36.916352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.006 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:36.916352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:36.916352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:36.916352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:36.916352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:36.916352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:36.916352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:36.916352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:36.916352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:36.916352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:36.916352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:36.916352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:36.916758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:36.916758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:36.916758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:36.916758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:36.916758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:36.916758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:36.916758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:36.916758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:36.916758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:36.916758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:36.916758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:36.916758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:36.916758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:36.917102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:36.917102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:36.917346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:36.917346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:36.917346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:36.917346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:36.917346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:36.917346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:36.917346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:36.917346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:36.917346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:36.917346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:36.917346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:36.917346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:36.917346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:36.917346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:36.917346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:36.917346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:36.917808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:36.918331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:36.918331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:36.918331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:36.980529 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:36.980797 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:36.980866 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:36.980976 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:42:36.982863 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:36.983044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:36.983480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:36.983480 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:36.983480 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:36.983480 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:36.983480 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:36.983480 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:36.983480 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:36.983480 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:36.984043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:36.984043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:36.984043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:36.984043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:36.984043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:36.984043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:36.984043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:36.984043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:36.984043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:36.984043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:36.984043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:36.984043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:36.984043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:36.984043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:36.984043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:36.984287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:36.984287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:36.984287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:36.984287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:36.984287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:36.984287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:36.984287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:36.984287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:36.984287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:36.984672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:36.984672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:36.984672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:36.984672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:36.984672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:36.984672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:36.984672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:36.984672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:42:36.984672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:36.984672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:36.984672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:36.984672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:36.984944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:36.984944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:36.984944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:36.984944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:36.984944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:36.984944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:36.984944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:36.984944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:36.984944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:36.984944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:36.984944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:36.984944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:36.984944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:36.984944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:36.985172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:36.985172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:36.985172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:36.985172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:36.985370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:36.985370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:36.985370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:36.985370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:36.985370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:36.985370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:36.985370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:36.985370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:36.985370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:36.985370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:36.985370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:36.985370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:36.985370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:36.985370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:36.985370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:36.985370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:36.985677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:36.985969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:36.985969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:36.985969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:36.985969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:36.985969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:36.985969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:36.985969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:36.999692 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:36.999941 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:37.000063 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:37.000197 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:37.000618 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:37.000845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.001336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:37.001336 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:37.001336 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:37.001336 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:37.001336 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:37.001336 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:37.001336 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.001336 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:37.002072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.002072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:37.002072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:37.002072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:37.002072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:37.002072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:37.002072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:37.002072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:37.002072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:37.002072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:37.002072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:37.002072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:37.002072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:37.002072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:37.002072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:37.002497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:37.002497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:37.002497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:37.002497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:37.002497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:37.002497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:37.002497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:37.002497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.002497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:37.002964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.002964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:37.002964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:37.002964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:37.002964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:37.002964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:37.002964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:37.002964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:37.002964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:37.002964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:37.002964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:37.003259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:37.003259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:37.003259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:37.003259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:37.003259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:37.003259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:37.003259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:37.003259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:37.003259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:37.003259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:37.003259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:37.003259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:37.003259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:37.003259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:37.003713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:37.003713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:37.003713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.003713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:37.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:37.004352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:37.004703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:37.004703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:37.004703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.004703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:37.024963 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:37.025171 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:37.025242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:37.028562 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:37.028562 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:37.028562 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:37.028562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.028562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:37.028562 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:37.028562 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:37.028562 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:37.028562 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:37.028562 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:37.028562 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.028562 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:37.028920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.028920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:37.028920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:37.028920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:37.028920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:37.028920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:37.028920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:37.028920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:37.028920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:37.028920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:37.028920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:37.028920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:37.029198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:37.029198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:37.029198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:37.029198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:37.029198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:37.029198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:37.029198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:37.029198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:37.029198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:37.029198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:37.029198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:37.029198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:37.029198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:37.029411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.029411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:37.029411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.029411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:37.029411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:37.029411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:37.029411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:37.029411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:37.029411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:37.029411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:37.029411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:37.029411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:37.029411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:37.029411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:37.029411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:37.029764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:37.030302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:37.030302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:37.030302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:37.030302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:37.030302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.030302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:37.050041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:37.050342 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:37.050455 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:37.052035 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:37.092786 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:37.092975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.094023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:37.094023 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:37.094023 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:37.094023 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:37.094023 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:37.094023 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:37.094023 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.094023 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:37.094747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.094747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:37.094747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:37.094747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:37.094747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:37.094747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:37.094747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:37.094747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:37.094747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:37.094747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:37.094747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:37.094747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:37.095134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:37.095134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:37.095134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:37.095134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:37.095134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:37.095134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:37.095134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:37.095134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:37.095134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:37.095134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:37.095134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:37.095134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:37.095134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:37.095519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.095519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:37.095519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.095519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:37.095519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:37.095519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:37.095519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:37.095519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:37.095519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:37.095519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:37.095519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:37.095519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:37.095519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:37.095519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:37.095519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:37.095973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:37.096433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:37.096433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:37.096433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:37.096433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:37.096433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:37.096433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:41.910273 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:41.910822 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:41.910822 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:41.911758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:41.912612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:41.912612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:41.912612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:41.912612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:41.912612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:41.912902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:41.912902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:41.912902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:41.913108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:41.913108 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:41.913108 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:41.913108 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:41.913108 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:41.913108 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:41.913108 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:41.913108 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:41.914054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:41.914054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:41.914054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:41.914054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:41.914054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:41.914054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:41.914054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:41.914054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:41.914054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:41.914054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:41.914054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:41.914054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:41.914355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:41.914355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:41.914355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:41.914355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:41.914355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:41.914355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:41.914355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:41.914355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:41.914355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:41.914355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:41.914355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:41.914355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:41.914355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:41.914623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:41.914623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:41.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:41.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:41.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:41.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:41.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:41.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:41.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:41.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:41.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:41.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:41.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:41.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:41.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:41.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:41.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:41.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:41.915458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:41.915865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:41.915865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:41.915865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:41.980912 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:41.981120 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:41.981210 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:41.981302 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:42:41.981686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:41.981852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:41.982260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:41.982260 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:41.982260 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:41.982260 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:41.982260 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:41.982260 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:41.982260 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:41.982260 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:41.982852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:41.982852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:41.982852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:41.982852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:41.982852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:41.982852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:41.982852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:41.982852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:41.982852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:41.982852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:41.982852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:41.982852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:41.982852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:41.982852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:41.982852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:41.983183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:41.983183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:41.983183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:41.983183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:41.983183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:41.983183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:41.983183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:41.983183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:41.983183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:41.983454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:41.983454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:41.983454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:41.983454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:41.983454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:41.983454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:41.983454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:41.983454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:42:41.983454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:41.983454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:41.983454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:41.983454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:41.983721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:41.983721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:41.983721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:41.983721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:41.983721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:41.983721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:41.983721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:41.983721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:41.983721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:41.983721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:41.983721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:41.983721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:41.983721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:41.983721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:41.983963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:41.983963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:41.983963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:41.983963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:41.984213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:41.984213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:41.984213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:41.984213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:41.984213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:41.984213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:41.984213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:41.984213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:41.984213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:41.984213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:41.984213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:41.984213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:41.984213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:41.984213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:41.984213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:41.984213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:41.984524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:41.984884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:41.984884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:41.984884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:41.984884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:41.984884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:41.984884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:41.984884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:42.000019 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:42.000196 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:42.000280 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:42.000380 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:42.000710 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:42.000941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.001423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:42.001423 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:42.001423 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:42.001423 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:42.001423 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:42.001423 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:42.001423 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.001423 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:42.002028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.002028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:42.002028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:42.002028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:42.002028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:42.002028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:42.002028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:42.002028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:42.002028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:42.002028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:42.002028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:42.002028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:42.002028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:42.002028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:42.002028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:42.002304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:42.002304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:42.002304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:42.002304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:42.002304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:42.002304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:42.002304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:42.002304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.002304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:42.002841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.002841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:42.002841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:42.002841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:42.002841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:42.002841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:42.002841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:42.002841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:42.002841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:42.002841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:42.002841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:42.003083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:42.003083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:42.003083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:42.003083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:42.003083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:42.003083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:42.003083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:42.003083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:42.003083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:42.003083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:42.003083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:42.003083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:42.003083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:42.003083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:42.003379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:42.003379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:42.003379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.003379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:42.004558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:42.004892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:42.005238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:42.005238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:42.005238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.005238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:42.025312 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:42.025508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:42.025649 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:42.025797 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:42.026078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:42.026301 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:42.026546 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.026960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:42.026960 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:42.026960 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:42.026960 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:42.026960 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:42.026960 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:42.026960 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.026960 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:42.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:42.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:42.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:42.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:42.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:42.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:42.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:42.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:42.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:42.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:42.027519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:42.027775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:42.027775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:42.027775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:42.027775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:42.027775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:42.027775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:42.027775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:42.027775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:42.027775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:42.027775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:42.027775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:42.027775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:42.027775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:42.027992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.027992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:42.028192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.028192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:42.028192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:42.028192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:42.028192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:42.028192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:42.028192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:42.028192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:42.028192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:42.028192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:42.028192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:42.028192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:42.028192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:42.028192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:42.028192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:42.028192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:42.028486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:42.028853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:42.028853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.028853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:42.050468 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:42.050764 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:42.050884 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:42.051698 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:42.090503 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:42.090712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.091219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:42.091219 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:42.091219 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:42.091219 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:42.091219 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:42.091219 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:42.091219 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.091219 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:42.091887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.091887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:42.091887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:42.091887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:42.091887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:42.091887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:42.091887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:42.091887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:42.091887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:42.091887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:42.091887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:42.091887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:42.092288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:42.092288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:42.092288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:42.092288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:42.092288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:42.092288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:42.092288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:42.092288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:42.092288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:42.092288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:42.092288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:42.092288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:42.092288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:42.092667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.092667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:42.092667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.092667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:42.092667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:42.092667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:42.092667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:42.092667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:42.092667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:42.092667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:42.092667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:42.092667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:42.092667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:42.092667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:42.092667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:42.093109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:42.093604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:42.093604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:42.093604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:42.093604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:42.093604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:42.093604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:46.910693 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:46.911133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:46.911133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:46.912965 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:46.913129 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:46.913250 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:46.913381 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:46.913517 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:46.913644 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:46.913777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:46.913894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:46.914011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:46.914527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:46.914527 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:46.914527 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:46.914527 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:46.914527 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:46.914527 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:46.914527 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:46.914527 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:46.915215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:46.915215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:46.915215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:46.915215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:46.915215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:46.915215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:46.915215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:46.915215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:46.915215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:46.915215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:46.915215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:46.915215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:46.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:46.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:46.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:46.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:46.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:46.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:46.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:46.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:46.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:46.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:46.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:46.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:46.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:46.915722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:46.915722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:46.915990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:46.915990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:46.915990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:46.915990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:46.915990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:46.915990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:46.915990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:46.915990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:46.915990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:46.915990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:46.915990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:46.915990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:46.915990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:46.915990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:46.915990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:46.915990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:46.916254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:46.916657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:46.916657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:46.916657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:46.982637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:46.982637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:46.982637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:46.982637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:42:46.982637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:46.982931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:46.984146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:46.984146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:46.984146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:46.984146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:46.984146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:46.984146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:46.984146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:46.984146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:46.984146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:46.984146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:46.984146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:46.984146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:46.984146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:46.984146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:46.984146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:46.984657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:46.984657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:46.984657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:46.984657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:46.984657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:46.984657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:46.984657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:46.984657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:46.984657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:46.984964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:46.984964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:46.984964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:46.984964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:46.984964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:46.984964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:46.984964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:46.984964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:42:46.984964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:46.984964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:46.984964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:46.984964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:46.985339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:46.985339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:46.985339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:46.985339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:46.985339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:46.985339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:46.985339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:46.985339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:46.985339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:46.985339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:46.985339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:46.985339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:46.985339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:46.985339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:46.985833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:46.985833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:46.985833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:46.985833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:46.985833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:46.985833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:46.985833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:46.985833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:46.985833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:46.985833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:46.985833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:46.985833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:46.985833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:46.985833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:46.986674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:46.987274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:46.987274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:46.987274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:46.987274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:46.987274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:46.987274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:46.987274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:46.987274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:46.987274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:46.987274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:46.987274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:46.987274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:47.003633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:47.003633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:47.003633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:47.003633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:47.003633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:47.003633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:47.003916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:47.004225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:47.004225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:47.004225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:47.004225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:47.004225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:47.004225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:47.004225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:47.004225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:47.004225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:47.004225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:47.004225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:47.004225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:47.004225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.004225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:47.004225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.004504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:47.004504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:47.004504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:47.004504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:47.004504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:47.004504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:47.004504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:47.004504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:47.004504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:47.004504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:47.004504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:47.004504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:47.004504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:47.004789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:47.004789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:47.004789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:47.004789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:47.004789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:47.004789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:47.004789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:47.004789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:47.004789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:47.004789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:47.004789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:47.004789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:47.004789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:47.005031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.005031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:47.005031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.005031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:47.005031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:47.005031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:47.005031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:47.005031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:47.005031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:47.005031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:47.005031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:47.005031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:47.005031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:47.005031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:47.005031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:47.005300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:47.005644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:47.005644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:47.005644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:47.005644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:47.005644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:47.005644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:47.005644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.005644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:47.025706 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:47.025911 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:47.026022 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:47.026175 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:47.026465 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:47.026684 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:47.026884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.027270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:47.027270 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:47.027270 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:47.027270 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:47.027270 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:47.027270 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:47.027270 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.027270 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:47.027985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.027985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:47.027985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:47.027985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:47.027985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:47.027985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:47.027985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:47.027985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:47.027985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:47.027985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:47.027985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:47.027985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:47.028260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:47.028260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:47.028260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:47.028260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:47.028260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:47.028260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:47.028260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:47.028260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:47.028260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:47.028260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:47.028260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:47.028260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:47.028260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:47.028512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.028512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:47.028882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.028882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:47.028882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:47.028882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:47.028882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:47.028882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:47.028882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:47.028882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:47.028882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:47.028882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:47.028882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:47.028882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:47.028882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:47.028882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:47.028882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:47.028882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:47.029172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:47.029517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:47.029517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.029517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:47.050872 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:47.051076 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:47.051205 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:47.051850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:47.089597 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:47.089781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.090114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:47.090114 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:47.090114 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:47.090114 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:47.090114 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:47.090114 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:47.090114 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.090114 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:47.090790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.090790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:47.090790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:47.090790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:47.090790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:47.090790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:47.090790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:47.090790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:47.090790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:47.090790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:47.090790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:47.090790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:47.091060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:47.091060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:47.091060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:47.091060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:47.091060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:47.091060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:47.091060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:47.091060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:47.091060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:47.091060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:47.091060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:47.091060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:47.091060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:47.091293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.091293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:47.091595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.091595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:47.091595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:47.091595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:47.091595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:47.091595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:47.091595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:47.091595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:47.091595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:47.091595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:47.091595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:47.091595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:47.091595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:47.091595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:47.091595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:47.091595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:47.091880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:47.092177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:47.092177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:47.092177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:51.911104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:51.911874 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:51.912020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:51.913136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:51.913293 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:51.913500 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:51.913695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a4cd3b09-db98-4b15-b4e9-a1e00d88974f, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:51.913816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:51.913916 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:51.914010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:51.914103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-154de3b1-a620-461a-833b-cf3c843c845b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:51.914197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:51.914683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:51.914683 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:51.914683 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:51.914683 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:51.914683 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:51.914683 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:51.914683 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:51.914683 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:51.915498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:51.915498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:51.915498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:51.915498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:51.915498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:51.915498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:51.915498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:51.915498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:51.915498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:51.915498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:51.915498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:51.915498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:51.916059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:51.916059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:51.916059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:51.916059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:51.916059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:51.916059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:51.916059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:51.916059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:51.916059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:51.916059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:51.916059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:51.916059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:51.916059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:51.916630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:51.916630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:51.916727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:51.916727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:51.916727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:51.916727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:51.916727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:51.916727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:51.916727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:51.916727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:51.916727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:51.916727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:51.916727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:51.916727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:51.916727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:51.916727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:51.916727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:51.916727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:51.917109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:51.917604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:51.917604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:51.917604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:51.982078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:51.982270 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:51.982365 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:51.982489 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:42:51.982836 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:51.982992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:51.983381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:51.983381 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:51.983381 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:51.983381 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:51.983381 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:51.983381 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:51.983381 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:51.983381 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:51.983965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:51.983965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:51.983965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:51.983965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:51.983965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:51.983965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:51.983965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:51.983965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:51.983965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:51.983965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:51.983965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:51.983965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:51.983965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:51.983965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:51.983965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:51.984529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:51.984529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:51.984529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:51.984529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:51.984529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:51.984529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:51.984529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:51.984529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:51.984529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:51.984842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:51.984842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:51.984842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:51.984842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:51.984842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:51.984842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:51.984842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:51.984842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:42:51.984842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:51.984842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:51.984842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:51.984842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:51.985191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:51.985191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:51.985191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:51.985191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:51.985191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:51.985191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:51.985191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:51.985191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:51.985191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:51.985191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:51.985191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:51.985191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:51.985191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:51.985191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:51.985996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:51.985996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:51.985996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:51.985996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:51.985996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:51.985996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:51.985996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:51.985996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:51.985996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:51.985996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:51.985996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:51.985996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:51.985996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:51.985996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:51.986383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:51.986740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:51.986740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:51.986740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:51.986740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:51.986740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:51.986740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:51.986740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:51.986740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:51.986740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:51.986740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:51.986740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:51.986740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:52.000773 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:52.001070 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:52.001198 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:52.001341 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:52.001929 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:52.002154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.002637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:52.002637 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:52.002637 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:52.002637 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:52.002637 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:52.002637 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:52.002637 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.002637 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:52.003483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.003483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:52.003483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:52.003483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:52.003483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:52.003483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:52.003483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:52.003483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:52.003483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:52.003483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:52.003483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:52.003483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:52.003483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:52.003483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:52.003483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:52.003836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:52.003836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:52.003836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:52.003836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:52.003836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:52.003836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:52.003836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:52.003836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.003836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:52.004477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.004477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:52.004477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:52.004477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:52.004477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:52.004477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:52.004477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:52.004477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:52.004477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:52.004477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:52.004477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:52.004875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:52.004875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:52.004875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:52.004875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:52.004875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:52.004875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:52.004875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:52.004875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:52.004875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:52.004875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:52.004875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:52.004875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:52.004875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:52.004875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:52.005320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:52.005320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:52.005320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.005320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:52.005683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:52.006253 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:52.006972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:52.006972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:52.006972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.006972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:52.026495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:52.026890 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:52.026981 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:52.027063 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:52.027495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:52.027776 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:52.028077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.028661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:52.028661 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:52.028661 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:52.028661 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:52.028661 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:52.028661 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:52.028661 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.028661 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:52.029303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.029303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:52.029303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:52.029303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:52.029303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:52.029303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:52.029303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:52.029303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:52.029303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:52.029303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:52.029303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:52.029303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:52.029785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:52.029785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:52.029785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:52.029785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:52.029785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:52.029785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:52.029785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:52.029785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:52.029785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:52.029785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:52.029785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:52.029785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:52.029785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:52.030160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.030160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:52.030160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.030160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:52.030160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:52.030160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:52.030160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:52.030160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:52.030160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:52.030160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:52.030160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:52.030160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:52.030160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:52.030160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:52.030160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:52.030662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:52.031125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:52.031125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:52.031125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:52.031125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:52.031125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.031125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:52.051276 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:52.051667 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:52.051758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:52.052319 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:52.081206 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:52.081467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.082017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:52.082017 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:52.082017 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:52.082017 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:52.082017 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:52.082017 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:52.082017 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.082017 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:52.082746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.082746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:52.082746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:52.082746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:52.082746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:52.082746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:52.082746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:52.082746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:52.082746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:52.082746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:52.082746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:52.082746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:52.083014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:52.083014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:52.083014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:52.083014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:52.083014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:52.083014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:52.083014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:52.083014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:52.083014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:52.083014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:52.083014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:52.083014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:52.083014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:52.083278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.083278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:52.083481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.083481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:52.083481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:52.083481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:52.083481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:52.083481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:52.083481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:52.083481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:52.083481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:52.083481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:52.083481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:52.083481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:52.083481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:52.083481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:52.083481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:52.083481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:52.083832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:52.084156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:52.084156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:52.084156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:56.911939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:56.912716 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:56.912716 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:56.913153 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:56.913320 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:56.913456 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:56.913964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:56.913964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:56.913964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:56.913964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:56.914340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:56.914340 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:56.914340 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:56.914340 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:56.914340 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:56.914340 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:56.914340 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:56.914340 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:56.915041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:56.915041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:56.915041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:56.915041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:56.915041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:56.915041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:56.915041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:56.915041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:56.915041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:56.915041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:56.915041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:56.915041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:56.916731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:56.916731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:56.916731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:56.916731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:56.916731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:56.916731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:56.916731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:56.916731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:56.916731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:56.916731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:56.916731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:56.916731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:56.916731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:56.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:56.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:56.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:56.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:56.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:56.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:56.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:56.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:56.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:56.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:56.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:56.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:56.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:56.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:56.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:56.918181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:56.918966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:56.918966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:56.918966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:56.918966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:56.918966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:56.918966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:56.982392 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:56.982832 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:56.982892 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:56.982978 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:42:56.983263 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:56.983438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:56.983888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:56.983888 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:56.983888 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:56.983888 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:56.983888 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:56.983888 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:56.983888 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:56.983888 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:56.984634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:56.984634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:56.984634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:56.984634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:56.984634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:56.984634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:56.984634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:56.984634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:56.984634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:56.984634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:56.984634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:56.984634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:56.984634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:56.984634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:56.984634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:56.985145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:56.985145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:56.985145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:56.985145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:56.985145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:56.985145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:56.985145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:56.985145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:56.985145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:56.985145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:56.985145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:56.985145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:56.985145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:56.985145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:56.985618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:56.985618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:56.985618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:42:56.985618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:56.985618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:56.985618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:56.985618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:56.985618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:56.985618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:56.985618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:56.985618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:56.985618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:56.985618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:56.985903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:56.985903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:56.985903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:56.985903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:56.985903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:56.985903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:56.985903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:56.985903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:56.985903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:56.985903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:56.985903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:56.985903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:56.986096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:56.986096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:56.986096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:56.986096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:56.986096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:56.986096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:56.986096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:56.986096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:56.986096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:56.986096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:56.986096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:56.986096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:56.986096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:56.986096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:56.986096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:56.986096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:56.986343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:56.986650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:56.986650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:56.986650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:56.986650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:56.986650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:56.986650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:56.986650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:57.001150 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:57.001533 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:57.001711 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:57.001838 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:57.002177 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:57.002377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.002874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:57.002874 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:57.002874 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:57.002874 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:57.002874 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:57.002874 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:57.002874 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.002874 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:57.003471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.003471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:42:57.003471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:57.003471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:57.003471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:57.003471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:57.003471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:57.003471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:42:57.003471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:57.003471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:42:57.003471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:57.003471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:42:57.003471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:57.003471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:42:57.003471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:42:57.003757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:57.003757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:42:57.003757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:57.003757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:42:57.003757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:42:57.003757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:57.003757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:42:57.003757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.003757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:42:57.004144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.004144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:57.004144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:57.004144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:57.004144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:57.004144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:57.004144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:57.004144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:57.004144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:57.004144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:57.004144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:57.004415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:57.004415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:57.004415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:57.004415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:57.004415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:57.004415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:57.004415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:57.004415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:57.004415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:57.004415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:57.004415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:57.004415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:57.004415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:57.004415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:57.004685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:57.004685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:57.004685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.004685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:57.005203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:57.005479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:57.005803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:57.005803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:57.005803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.005803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:57.026855 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:57.027222 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:57.027341 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:57.027510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:57.027790 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:57.027978 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:57.028151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.029439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:57.029439 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:57.029439 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:57.029439 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:57.029439 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:57.029439 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:57.029439 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.029439 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:57.030058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.030058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:57.030058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:57.030058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:57.030058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:57.030058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:57.030058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:57.030058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:57.030058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:57.030058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:57.030058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:57.030058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:57.030255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:57.030255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:57.030255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:57.030255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:57.030255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:57.030255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:57.030255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:57.030255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:57.030255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:57.030255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:57.030255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:57.030255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:57.030255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:57.030502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.030502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:57.031584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.031584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:57.031584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:57.031584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:57.031584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:57.031584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:57.031584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:57.031584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:57.031584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:57.031584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:57.031584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:57.031584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:57.031584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:57.031584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:57.031584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:57.031584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:57.031883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:57.032156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:57.032156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.032156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:42:57.051684 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:42:57.052050 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:42:57.052164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:42:57.052881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:42:57.093054 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:42:57.093324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.094237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:42:57.094237 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:57.094237 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:42:57.094237 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:42:57.094237 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:57.094237 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:42:57.094237 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.094237 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:42:57.095552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.095552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:42:57.095552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:57.095552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:42:57.095552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:57.095552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:57.095552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:57.095552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:57.095552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:57.095552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:57.095552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:42:57.095552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:57.095886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:42:57.095886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:57.095886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:42:57.095886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:57.095886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:42:57.095886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:42:57.095886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:57.095886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:42:57.095886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:57.095886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:42:57.095886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:42:57.095886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:57.095886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:42:57.096092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.096092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:42:57.096640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.096640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:42:57.096640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:42:57.096640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:42:57.096640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:42:57.096640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:42:57.096640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:57.096640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:42:57.096640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:42:57.096640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:57.096640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:42:57.096640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:42:57.096640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:42:57.096640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:42:57.096640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:42:57.096640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:42:57.096946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:42:57.097223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:42:57.097223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:42:57.097223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:01.912480 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:01.913032 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:01.913084 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:01.914003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:01.914207 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:01.914421 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:01.914640 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:01.914806 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:01.914950 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:01.915091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:01.915565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:01.915565 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:01.915565 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:01.915565 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:01.915565 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:01.915565 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:01.915565 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:01.915565 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:01.916368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:01.916368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:01.916368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:01.916368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:01.916368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:01.916368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:01.916368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:01.916368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:01.916368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:01.916368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:01.916368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:01.916368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:01.916681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:01.916681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:01.916681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:01.916681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:01.916681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:01.916681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:01.916681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:01.916681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:01.916681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:01.916681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:01.916681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:01.916681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:01.916681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:01.916995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:01.916995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:01.917657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:01.917657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:01.917657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:01.917657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:01.917657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:01.917657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:01.917657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:01.917657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:01.917657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:01.917657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:01.917657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:01.917657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:01.917657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:01.917657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:01.917657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:01.917657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:01.918092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:01.918476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:01.918476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:01.918476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:01.982828 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:01.985669 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:01.986064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:01.986654 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:43:01.987226 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:01.987427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:01.988112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:01.988112 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:01.988112 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:01.988112 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:01.988112 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:01.988112 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:01.988112 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:01.988112 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:01.988839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:01.988839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:01.988839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:01.988839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:01.988839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:01.988839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:01.988839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:01.988839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:01.988839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:01.988839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:01.988839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:01.988839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:01.988839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:01.988839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:01.988839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:01.989197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:01.989197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:01.989197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:01.989197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:01.989197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:01.989197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:01.989197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:01.989197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:01.989197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:01.991067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:01.991067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:01.991067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:01.991067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:01.991067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:01.991067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:01.991067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:01.991067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:43:01.991067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:01.991067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:01.991067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:01.991067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:01.991329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:01.991329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:01.991329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:01.991329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:01.991329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:01.991329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:01.991329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:01.991329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:01.991329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:01.991329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:01.991329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:01.991329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:01.991329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:01.991329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:01.991602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:01.991602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:01.991602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:01.991602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:01.993476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:01.993476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:01.993476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:01.993476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:01.993476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:01.993476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:01.993476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:01.993476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:01.993476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:01.993476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:01.993476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:01.993476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:01.993476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:01.993476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:01.993476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:01.993476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:01.993918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:01.994322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:01.994322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:01.994322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:01.994322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:01.994322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:01.994322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:01.994322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:02.001643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:02.002041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:02.002170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:02.002304 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:02.002741 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:02.005318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.005829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:02.005829 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:02.005829 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:02.005829 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:02.005829 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:02.005829 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:02.005829 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.005829 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:02.006548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.006548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:02.006548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:02.006548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:02.006548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:02.006548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:02.006548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:02.006548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:02.006548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:02.006548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:02.006548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:02.006548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:02.006548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:02.006548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:02.006548 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:02.006832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:02.006832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:02.006832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:02.006832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:02.006832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:02.006832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:02.006832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:02.006832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.006832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:02.008038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.008038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:02.008038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:02.008038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:02.008038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:02.008038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:02.008038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:02.008038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:02.008038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:02.008038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:02.008038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:02.008289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:02.008289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:02.008289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:02.008289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:02.008289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:02.008289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:02.008289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:02.008289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:02.008289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:02.008289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:02.008289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:02.008289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:02.008289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:02.008289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:02.008635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:02.008635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:02.008635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.008635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:02.009676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:02.010207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:02.010991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:02.010991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:02.010991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.010991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:02.027255 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:02.027554 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:02.027662 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:02.027794 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:02.028088 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:02.028352 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:02.028628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.029058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:02.029058 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:02.029058 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:02.029058 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:02.029058 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:02.029058 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:02.029058 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.029058 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:02.029808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.029808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:02.029808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:02.029808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:02.029808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:02.029808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:02.029808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:02.029808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:02.029808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:02.029808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:02.029808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:02.029808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:02.030060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:02.030060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:02.030060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:02.030060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:02.030060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:02.030060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:02.030060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:02.030060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:02.030060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:02.030060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:02.030060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:02.030060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:02.030060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:02.030496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.030496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:02.031105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.031105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:02.031105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:02.031105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:02.031105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:02.031105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:02.031105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:02.031105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:02.031105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:02.031105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:02.031105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:02.031105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:02.031105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:02.031105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:02.031105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:02.031105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:02.031436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:02.031866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:02.031866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.031866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:02.052113 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:02.052345 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:02.052471 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:02.053076 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:02.085217 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:02.088872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.089331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:02.089331 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:02.089331 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:02.089331 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:02.089331 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:02.089331 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:02.089331 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.089331 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:02.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:02.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:02.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:02.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:02.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:02.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:02.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:02.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:02.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:02.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:02.090245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:02.090533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:02.090533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:02.090533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:02.090533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:02.090533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:02.090533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:02.090533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:02.090533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:02.090533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:02.090533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:02.090533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:02.090533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:02.090533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:02.090749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.090749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:02.091762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.091762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:02.091762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:02.091762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:02.091762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:02.091762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:02.091762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:02.091762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:02.091762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:02.091762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:02.091762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:02.091762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:02.091762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:02.091762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:02.091762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:02.091762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:02.092107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:02.092434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:02.092434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:02.092434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:06.913077 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:06.913635 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:06.913635 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:06.914212 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:06.914398 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:06.914536 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:06.914658 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:06.914759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:06.914860 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:06.914965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:06.915383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:06.915383 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:06.915383 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:06.915383 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:06.915383 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:06.915383 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:06.915383 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:06.915383 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:06.916007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:06.916007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:06.916007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:06.916007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:06.916007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:06.916007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:06.916007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:06.916007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:06.916007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:06.916007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:06.916007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:06.916007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:06.916474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:06.916474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:06.916474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:06.916474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:06.916474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:06.916474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:06.916474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:06.916474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:06.916474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:06.916474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:06.916474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:06.916474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:06.916474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:06.916956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:06.916956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:06.916956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:06.916956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:06.916956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:06.916956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:06.916956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:06.916956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:06.916956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:06.916956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:06.916956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:06.916956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:06.916956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:06.916956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:06.916956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:06.917477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:06.917889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:06.917889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:06.917889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:06.917889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:06.917889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:06.917889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:06.985295 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:06.986411 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:06.987017 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:06.987356 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:43:06.987760 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:06.987960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:06.988863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:06.988863 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:06.988863 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:06.988863 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:06.988863 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:06.988863 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:06.988863 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:06.988863 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:06.989646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:06.989646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:06.989646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:06.989646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:06.989646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:06.989646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:06.989646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:06.989646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:06.989646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:06.989646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:06.989646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:06.989646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:06.989646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:06.989646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:06.989646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:06.989979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:06.989979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:06.989979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:06.989979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:06.989979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:06.989979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:06.989979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:06.989979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:06.989979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:06.990511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:06.990511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:06.990511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:06.990511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:06.990511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:06.990511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:06.990511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:06.990511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:43:06.990511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:06.990511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:06.990511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:06.990511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:06.991127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:06.991127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:06.991127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:06.991127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:06.991127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:06.991127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:06.991127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:06.991127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:06.991127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:06.991127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:06.991127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:06.991127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:06.991127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:06.991127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:06.991867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:06.991867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:06.991867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:06.991867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:06.991867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:06.991867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:06.991867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:06.991867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:06.991867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:06.991867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:06.991867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:06.991867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:06.991867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:06.991867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:06.992286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:06.992784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:06.992784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:06.992784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:06.992784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:06.992784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:06.992784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:06.992784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:06.992784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:06.992784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:06.992784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:06.992784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:06.992784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:07.002097 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:07.003863 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:07.004045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:07.006119 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:07.007897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:07.008150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.008708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:07.008708 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:07.008708 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:07.008708 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:07.008708 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:07.008708 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:07.008708 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.008708 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:07.009313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.009313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:07.009313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:07.009313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:07.009313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:07.009313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:07.009313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:07.009313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:07.009313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:07.009313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:07.009313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:07.009313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:07.009313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:07.009313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:07.009313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:07.009612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:07.009612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:07.009612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:07.009612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:07.009612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:07.009612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:07.009612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:07.009612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.009612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:07.010034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.010034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:07.010034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:07.010034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:07.010034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:07.010034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:07.010034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:07.010034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:07.010034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:07.010034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:07.010034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:07.010305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:07.010305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:07.010305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:07.010305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:07.010305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:07.010305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:07.010305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:07.010305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:07.010305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:07.010305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:07.010305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:07.010305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:07.010305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:07.010305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:07.010755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:07.010755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:07.010755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.010755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:07.011159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:07.011656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:07.012131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:07.012131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:07.012131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.012131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:07.027790 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:07.029336 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:07.029336 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:07.031114 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:07.033715 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:07.034063 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:07.034364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.037245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:07.037245 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:07.037245 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:07.037245 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:07.037245 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:07.037245 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:07.037245 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.037245 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:07.037924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.037924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:07.037924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:07.037924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:07.037924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:07.037924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:07.037924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:07.037924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:07.037924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:07.037924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:07.037924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:07.037924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:07.038411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:07.038411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:07.038411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:07.038411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:07.038411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:07.038411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:07.038411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:07.038411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:07.038411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:07.038411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:07.038411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:07.038411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:07.038411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:07.038861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.038861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:07.039165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.039165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:07.039165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:07.039165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:07.039165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:07.039165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:07.039165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:07.039165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:07.039165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:07.039165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:07.039165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:07.039165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:07.039165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:07.039165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:07.039165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:07.039165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:07.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:07.040199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:07.040199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.040199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:07.053567 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:07.054016 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:07.054131 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:07.054933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:07.104937 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:07.105055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.105483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:07.105483 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:07.105483 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:07.105483 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:07.105483 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:07.105483 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:07.105483 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.105483 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:07.106225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.051 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.106225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:07.106225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:07.106225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:07.106225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:07.106225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:07.106225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:07.106225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:07.106225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:07.106225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:07.106225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:07.106225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:07.106638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:07.106638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:07.106638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:07.106638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:07.106638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:07.106638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:07.106638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:07.106638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:07.106638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:07.106638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:07.106638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:07.106638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:07.106638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:07.106965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.106965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:07.107093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.107093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:07.107093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:07.107093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:07.107093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:07.107093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:07.107093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:07.107093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:07.107093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:07.107093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:07.107093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:07.107093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:07.107093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:07.107093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:07.107093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:07.107093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:07.107619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:07.108160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:07.108160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:07.108160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:11.913596 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:11.914173 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:11.914173 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:11.914581 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:11.914751 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:11.915018 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:11.915018 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:11.915122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:11.915171 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:11.915290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:11.915734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:11.915734 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:11.915734 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:11.915734 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:11.915734 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:11.915734 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:11.915734 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:11.915734 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:11.916328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:11.916328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:11.916328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:11.916328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:11.916328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:11.916328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:11.916328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:11.916328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:11.916328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:11.916328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:11.916328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:11.916328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:11.916730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:11.916730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:11.916730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:11.916730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:11.916730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:11.916730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:11.916730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:11.916730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:11.916730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:11.916730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:11.916730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:11.916730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:11.916730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:11.917122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:11.917122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:11.917122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:11.917122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:11.917122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:11.917122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:11.917122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:11.917122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:11.917122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:11.917122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:11.917122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:11.917122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:11.917122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:11.917122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:11.917122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:11.917621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:11.918169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:11.918169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:11.918169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:11.918169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:11.918169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:11.918169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:11.985829 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:11.986144 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:11.986258 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:11.986358 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:43:11.986894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:11.987075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:11.987568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:11.987568 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:11.987568 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:11.987568 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:11.987568 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:11.987568 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:11.987568 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:11.987568 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:11.988195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:11.988195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:11.988195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:11.988195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:11.988195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:11.988195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:11.988195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:11.988195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:11.988195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:11.988195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:11.988195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:11.988195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:11.988195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:11.988195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:11.988195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:11.988554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:11.988554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:11.988554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:11.988554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:11.988554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:11.988554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:11.988554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:11.988554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:11.988554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:11.988836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:11.988836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:11.988836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:11.988836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:11.988836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:11.988836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:11.988836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:11.988836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:43:11.988836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:11.988836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:11.988836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:11.988836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:11.989065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:11.989065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:11.989065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:11.989065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:11.989065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:11.989065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:11.989065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:11.989065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:11.989065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:11.989065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:11.989065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:11.989065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:11.989065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:11.989065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:11.989290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:11.989290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:11.989290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:11.989290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:11.989579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:11.989579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:11.989579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:11.989579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:11.989579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:11.989579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:11.989579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:11.989579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:11.989579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:11.989579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:11.989579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:11.989579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:11.989579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:11.989579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:11.989579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:11.989579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:11.989872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:11.990166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:11.990166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:11.990166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:11.990166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:11.990166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:11.990166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:11.990166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:12.002628 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:12.003128 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:12.003246 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:12.003365 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:12.005023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:12.005288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.005835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:12.005835 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:12.005835 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:12.005835 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:12.005835 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:12.005835 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:12.005835 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.005835 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:12.006473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.006473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:12.006473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:12.006473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:12.006473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:12.006473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:12.006473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:12.006473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:12.006473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:12.006473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:12.006473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:12.006473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:12.006473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:12.006473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:12.006473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:12.006771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:12.006771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:12.006771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:12.006771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:12.006771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:12.006771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:12.006771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:12.006771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.006771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:12.007111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.007111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:12.007111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:12.007111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:12.007111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:12.007111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:12.007111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:12.007111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:12.007111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:12.007111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:12.007111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:12.007334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:12.007334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:12.007334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:12.007334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:12.007334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:12.007334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:12.007334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:12.007334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:12.007334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:12.007334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:12.007334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:12.007334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:12.007334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:12.007334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:12.007593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:12.007593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:12.007593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.007593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:12.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:12.008170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:12.008493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:12.008493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:12.008493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.008493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:12.030179 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:12.031196 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:12.031284 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:12.031426 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:12.031875 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:12.032098 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:12.032312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.033035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:12.033035 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:12.033035 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:12.033035 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:12.033035 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:12.033035 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:12.033035 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.033035 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:12.033991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.033991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:12.033991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:12.033991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:12.033991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:12.033991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:12.033991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:12.033991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:12.033991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:12.033991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:12.033991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:12.033991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:12.034638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:12.034638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:12.034638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:12.034638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:12.034638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:12.034638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:12.034638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:12.034638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:12.034638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:12.034638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:12.034638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:12.034638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:12.034638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:12.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:12.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:12.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:12.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:12.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:12.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:12.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:12.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:12.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:12.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:12.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:12.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:12.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:12.035552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:12.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:12.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:12.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:12.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:12.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:12.055404 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:12.057156 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:12.057156 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:12.057156 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:12.100960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:12.101122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.101770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:12.101770 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:12.101770 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:12.101770 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:12.101770 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:12.101770 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:12.101770 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.101770 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:12.104400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.104400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:12.104400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:12.104400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:12.104400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:12.104400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:12.104400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:12.104400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:12.104400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:12.104400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:12.104400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:12.104400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:12.104705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:12.104705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:12.104705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:12.104705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:12.104705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:12.104705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:12.104705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:12.104705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:12.104705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:12.104705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:12.104705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:12.104705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:12.104705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:12.104980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.104980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:12.104980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.104980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:12.104980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:12.104980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:12.104980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:12.104980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:12.104980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:12.104980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:12.104980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:12.104980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:12.104980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:12.104980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:12.104980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:12.105493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:12.105851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:12.105851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:12.105851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:12.105851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:12.105851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:12.105851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:16.913933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:16.915328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:16.915328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:16.915328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:16.915328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:16.915469 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:16.915527 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:16.915665 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:16.915767 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:16.915865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:16.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:16.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:16.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:16.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:16.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:16.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:16.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:16.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:16.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:16.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:16.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:16.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:16.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:16.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:16.917622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:16.917920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:16.917920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:16.917920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:16.917920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:16.917920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:16.917920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:16.917920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:16.917920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:16.917920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:16.917920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:16.917920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:16.917920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:16.917920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:16.917920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:16.918171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:16.918171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:16.918171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:16.918171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:16.918171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:16.918171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:16.918171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:16.918171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:16.918171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:16.918171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:16.918171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:16.918171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:16.918171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:16.918171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:16.918417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:16.918828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:16.918828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:16.918828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:16.918828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:16.918828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:16.918828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:16.918828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:16.918828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:16.918828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:16.918828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:16.986243 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:16.986893 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:16.986893 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:16.986893 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:43:16.987149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:16.987353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:16.989858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:16.990399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:16.990399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:16.990399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:16.990399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:16.990399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:16.990399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:16.990399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:16.990399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:16.990399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:16.990399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:16.990399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:16.990399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:16.990399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:16.990399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:16.990399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:16.990928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:16.990928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:16.990928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:16.990928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:16.990928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:16.990928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:16.990928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:43:16.990928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:16.990928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:16.990928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:16.990928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:16.990928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:16.990928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:16.991275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:16.991275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:16.991275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:16.991275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:16.991275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:16.991275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:16.991275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:16.991275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:16.991275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:16.991275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:16.991275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:16.991275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:16.991275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:16.991275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:16.991648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:16.991648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:16.991648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:16.991648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:16.991648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:16.991648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:16.991648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:16.991648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:16.991648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:16.991648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:16.991648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:16.991648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:16.991648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:16.991648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:16.991648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:16.992042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:16.992581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:16.992581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:16.992581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:16.992581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:16.992581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:16.992581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:16.992581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:16.992581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:16.992581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:17.003160 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:17.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:17.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:17.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:17.004060 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:17.004276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:17.006664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:17.007242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:17.007242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:17.007242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:17.007242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:17.007242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:17.007242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:17.007242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:17.007242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:17.007242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:17.007242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:17.007242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:17.007242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:17.007242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.007242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:17.007242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.007851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:17.007851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:17.007851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:17.007851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:17.007851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:17.007851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:17.007851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:17.007851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:17.007851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:17.007851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:17.007851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:17.007851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:17.007851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:17.008239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:17.008239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:17.008239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:17.008239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:17.008239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:17.008239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:17.008239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:17.008239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:17.008239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:17.008239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:17.008239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:17.008239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:17.008239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:17.008667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.008667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:17.008667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.008667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:17.008667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:17.008667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:17.008667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:17.008667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:17.008667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:17.008667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:17.008667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:17.008667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:17.008667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:17.008667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:17.008667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:17.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:17.009655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:17.009655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:17.009655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:17.009655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:17.009655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:17.009655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:17.009655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.009655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:17.031267 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:17.032101 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:17.032101 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:17.032101 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:17.032337 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:17.032553 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:17.032742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.035264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:17.035264 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:17.035264 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:17.035264 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:17.035264 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:17.035264 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:17.035264 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.035264 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:17.035264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.035264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:17.035264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:17.035264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:17.035264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:17.035264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:17.035264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:17.035868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:17.035868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:17.035868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:17.035868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:17.035868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:17.035868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:17.035868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:17.035868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:17.035868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:17.035868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:17.035868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:17.035868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:17.035868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:17.035868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:17.036298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:17.036298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:17.036298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:17.036298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:17.036298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.036298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:17.036298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.036298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:17.036298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:17.036298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:17.036298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:17.036298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:17.036298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:17.036298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:17.036738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:17.037381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:17.037381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:17.037381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:17.037381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:17.037381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:17.037381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:17.037381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:17.037381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:17.037381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.037381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:17.055939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:17.057922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:17.057922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:17.057922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:17.096622 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:17.096798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.098572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:17.098572 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:17.098572 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:17.098572 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:17.098572 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:17.098572 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:17.098572 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.098572 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:17.098572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.041 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.098572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:17.098572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:17.098572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:17.098572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:17.098572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:17.098572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:17.099111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:17.099111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:17.099111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:17.099111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:17.099111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:17.099111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:17.099111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:17.099111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:17.099111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:17.099111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:17.099111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:17.099111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:17.099111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:17.099111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:17.099531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:17.099531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:17.099531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:17.099531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:17.099531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.099531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:17.100709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.100709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:17.100709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:17.100709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:17.100709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:17.100709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:17.100709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:17.100709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:17.100709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:17.100709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:17.100709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:17.100709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:17.100709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:17.100709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:17.100709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:17.100709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:17.101192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:17.101711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:17.101711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:17.101711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:21.914462 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:21.916104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:21.916104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:21.916557 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:21.916832 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:21.917007 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:21.917153 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:21.917273 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:21.917392 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:21.917524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:21.918575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:21.918575 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:21.918575 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:21.918575 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:21.918575 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:21.918575 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:21.918575 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:21.918575 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:21.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:21.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:21.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:21.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:21.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:21.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:21.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:21.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:21.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:21.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:21.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:21.919241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:21.919602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:21.919602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:21.919602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:21.919602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:21.919602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:21.919602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:21.919602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:21.919602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:21.919602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:21.919602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:21.919602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:21.919602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:21.919602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:21.919869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:21.919869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:21.920864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:21.921231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:21.921660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:21.921660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:21.921660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:21.986881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:21.987825 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:21.987825 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:21.987825 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:43:21.988185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:21.988361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:21.991145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:21.991684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:21.991684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:21.991684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:21.991684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:21.991684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:21.991684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:21.991684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:21.991684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:21.991684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:21.991684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:21.991684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:21.991684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:21.991684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:21.991684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:21.991684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:21.991975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:21.991975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:21.991975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:21.991975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:21.991975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:21.991975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:21.991975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:43:21.991975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:21.991975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:21.991975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:21.991975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:21.991975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:21.991975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:21.992304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:21.992304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:21.992304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:21.992304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:21.992304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:21.992304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:21.992304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:21.992304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:21.992304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:21.992304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:21.992304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:21.992304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:21.992304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:21.992304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:21.992593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:21.992593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:21.992593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:21.992593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:21.992593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:21.992593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:21.992593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:21.992593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:21.992593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:21.992593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:21.992593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:21.992593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:21.992593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:21.992593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:21.992593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:21.992841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:21.993235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:21.993235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:21.993235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:21.993235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:21.993235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:21.993235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:21.993235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:21.993235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:21.993235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:22.003752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:22.004400 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:22.004400 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:22.004400 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:22.004681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:22.004889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:22.007393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:22.007884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:22.007884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:22.007884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:22.007884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:22.007884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:22.007884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:22.007884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:22.007884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:22.007884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:22.007884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:22.007884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:22.007884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:22.007884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.007884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:22.007884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.008177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:22.008177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:22.008177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:22.008177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:22.008177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:22.008177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:22.008177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:22.008177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:22.008177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:22.008177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:22.008177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:22.008177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:22.008177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:22.008414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:22.008414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:22.008414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:22.008414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:22.008414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:22.008414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:22.008414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:22.008414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:22.008414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:22.008414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:22.008414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:22.008414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:22.008414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:22.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:22.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:22.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:22.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:22.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:22.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:22.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:22.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:22.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:22.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:22.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:22.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:22.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:22.008963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:22.009289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:22.009289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:22.009289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:22.009289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:22.009289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:22.009289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:22.009289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.009289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:22.031841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:22.032639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:22.032639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:22.032639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:22.032880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:22.033065 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:22.033232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.033672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:22.033672 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:22.033672 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:22.033672 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:22.033672 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:22.033672 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:22.033672 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.033672 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:22.036236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.036236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:22.036236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:22.036236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:22.036236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:22.036236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:22.036236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:22.036236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:22.036236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:22.036236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:22.036236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:22.036236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:22.036739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:22.036739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:22.036739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:22.036739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:22.036739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:22.036739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:22.036739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:22.036739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:22.036739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:22.036739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:22.036739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:22.036739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:22.036739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:22.037130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.037130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:22.037130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.037130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:22.037130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:22.037130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:22.037130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:22.037130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:22.037130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:22.037130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:22.037130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:22.037130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:22.037130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:22.037130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:22.037130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:22.037642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:22.038228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:22.038228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:22.038228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:22.038228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:22.038228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.038228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:22.056488 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:22.058530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:22.058530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:22.058530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:22.065387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=239 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:43:22.102128 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:22.102369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.103029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:22.103029 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:22.103029 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:22.103029 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:22.103029 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:22.103029 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:22.103029 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.103029 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:22.104957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.046 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.104957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:22.104957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:22.104957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:22.104957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:22.104957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:22.104957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:22.104957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:22.104957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:22.104957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:22.104957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:22.104957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:22.105386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:22.105386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:22.105386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:22.105386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:22.105386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:22.105386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:22.105386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:22.105386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:22.105386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:22.105386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:22.105386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:22.105386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:22.105386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:22.105807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.105807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:22.105807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.105807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:22.105807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:22.105807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:22.105807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:22.105807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:22.105807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:22.105807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:22.105807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:22.105807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:22.105807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:22.105807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:22.105807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:22.106260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:22.106825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:22.106825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:22.106825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:22.106825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:22.106825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:22.106825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:26.915496 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:26.916988 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:26.917170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:26.918174 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:26.918402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:26.918603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:26.918745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:26.918855 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:26.918963 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:26.919092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:26.919202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:26.921393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:26.921393 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:26.921393 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:26.921393 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:26.921393 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:26.921393 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:26.921393 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:26.921393 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:26.921393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:26.921393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:26.921393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:26.921393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:26.921393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:26.921393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:26.921393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:26.921837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:26.921837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:26.921837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:26.921837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:26.921837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:26.921837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:26.921837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:26.921837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:26.921837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:26.921837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:26.921837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:26.921837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:26.921837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:26.921837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:26.922196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:26.922196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:26.922196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:26.922196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:26.922196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:26.922196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:26.922196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:26.922196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:26.922196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:26.922196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:26.922196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:26.922196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:26.922196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:26.922196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:26.922478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:26.922812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:26.922812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:26.922812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:26.922812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:26.922812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:26.922812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:26.922812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:26.922812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:26.922812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:26.922812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:26.987440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:26.988405 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:26.988405 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:26.988405 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:43:26.988674 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:26.988840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:26.989822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:26.989822 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:26.989822 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:26.989822 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:26.989822 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:26.989822 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:26.989822 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:26.989822 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:26.991336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:26.991336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:26.991336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:26.991336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:26.991336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:26.991336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:26.991336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:26.991336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:26.991336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:26.991336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:26.991336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:26.991336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:26.991336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:26.991336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:26.991336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:26.993159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:26.993159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:26.993159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:26.993159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:26.993159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:26.993159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:26.993159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:26.993159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:26.993159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:26.993159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:26.993159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:26.993159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:26.993159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:26.993159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:26.993472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:26.993472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:26.993472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:43:26.993472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:26.993472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:26.993472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:26.993472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:26.993472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:26.993472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:26.993472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:26.993472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:26.993472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:26.993472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:26.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:26.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:26.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:26.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:26.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:26.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:26.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:26.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:26.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:26.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:26.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:26.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:26.995737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:26.995737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:26.995737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:26.995737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:26.995737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:26.995737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:26.995737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:26.995737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:26.995737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:26.995737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:26.995737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:26.995737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:26.995737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:26.995737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:26.995737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:26.995737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:26.996916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:26.997811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:27.004177 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:27.005028 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:27.005028 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:27.005028 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:27.005290 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.005491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.005957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.005957 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.005957 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.005957 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.005957 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.005957 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.005957 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.005957 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.007584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.007584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:27.007584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:27.007584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:27.007584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.007584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:27.007584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.007584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.007584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.007584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:27.007584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.007584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:27.007584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.007584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:27.007584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:27.008676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.008676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:27.008676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.008676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:27.008676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:27.008676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.008676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:27.008676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.008676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:27.009150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.009150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.009150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:27.009150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:27.009150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:27.009150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:27.009150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:27.009150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:27.009150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:27.009150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.009150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.010347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.010347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.010347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.010347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.010347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.010347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.010347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.010347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.010347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.010347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.010347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.010347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.010347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.010347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.010632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.010632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.010632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.010632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:27.011595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:27.012327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:27.012649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.012649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:27.012649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.012649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:27.032359 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:27.033142 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:27.033142 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:27.033142 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:27.033422 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.033660 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.033844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.034309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.034309 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.034309 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.034309 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.034309 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.034309 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.034309 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.034309 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.036219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.036219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.036219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:27.036219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:27.036219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:27.036219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:27.036219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.036219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.036219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.036219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.036219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.036219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.036437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.036437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.036437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.036437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.036437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.036437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.036437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.036437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.036437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.036437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.036437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.036437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.036437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.036684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.036684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.036684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.036684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:27.036684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:27.036684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:27.036684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:27.036684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:27.036684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:27.036684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:27.036684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:27.036684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:27.036684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:27.036684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:27.036684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:27.036916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.037205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:27.037205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:27.037205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.037205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:27.037205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.037205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:27.057010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:27.058756 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:27.058756 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:27.058756 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:27.075479 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:27.090133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.090519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.093391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.093391 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.093391 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.093391 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.093391 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.093391 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.093391 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.093391 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.093391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.093391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.093391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:27.093391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:27.093391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:27.093391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:27.093391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.093773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.093773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.093773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.093773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.093773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.093773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.093773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.093773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.093773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.093773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.093773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.093773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.093773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.093773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.093990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.093990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.093990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.093990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.093990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.093990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.093990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.093990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:27.093990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:27.093990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:27.093990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:27.093990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:27.093990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:27.093990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:27.094262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:27.094620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.094620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:27.094620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.094620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:27.094620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:27.094620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.094620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:27.094620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.094620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:27.094620 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:27.094620 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:27.103777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 01 11:43:27.104131 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Number of inconsistencies found at create/update: networks=8, security_groups=1, routers=5, router_ports=5 {{(pid=89427) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 01 11:43:27.104484 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=13 {{(pid=89427) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 01 11:43:27.104703 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 0d4134e5-3b3a-49d4-a466-c1b6f993b0ab (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:27.136433 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-0d4134e5-3b3a-49d4-a466-c1b6f993b0ab, resource={'id': '0d4134e5-3b3a-49d4-a466-c1b6f993b0ab', 'name': 'private', 'tenant_id': '43b6be50ae544caeb6a73eced65fb6a7', 'project_id': '43b6be50ae544caeb6a73eced65fb6a7', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['16527f37-b4ae-44e5-9382-abb5e59d1cad', 'a33d59fc-0fd4-45c5-9fff-82704a1922a6'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:23:10Z', 'updated_at': '2026-06-01T11:23:13Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 48058}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.136818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-0d4134e5-3b3a-49d4-a466-c1b6f993b0ab, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.137120 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-bedd3a6a-831e-4f6a-adf7-0842bd968bac, external_ids_update=None, columns={'tag': 48058}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.137229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.139501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.139501 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.139501 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.139501 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.139501 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.139501 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.139501 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.139501 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.139501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 0d4134e5-3b3a-49d4-a466-c1b6f993b0ab (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.139501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.139501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:27.139501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:27.139501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:27.139501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:27.140333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:43:27.140333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.140333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.140333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.140333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.140333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.140333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.140333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.140333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.140333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.140333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.140333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.140333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.140994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.140994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.140994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.140994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.140994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.140994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.140994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.140994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.140994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.140994 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 19da64d5-62e3-4b38-84f1-7d0971806ce9 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:27.169642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-19da64d5-62e3-4b38-84f1-7d0971806ce9, resource={'id': '19da64d5-62e3-4b38-84f1-7d0971806ce9', 'name': 'shared', 'tenant_id': '43b6be50ae544caeb6a73eced65fb6a7', 'project_id': '43b6be50ae544caeb6a73eced65fb6a7', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['c855657f-8ba2-422e-88fd-c5a71ec03ed7'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:24:23Z', 'updated_at': '2026-06-01T11:24:23Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 19987}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.170084 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-19da64d5-62e3-4b38-84f1-7d0971806ce9, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.170399 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-cb9dd134-19dd-491b-aafb-e4edb81fcf60, external_ids_update=None, columns={'tag': 19987}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.170525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.172646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.172646 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.172646 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.172646 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.172646 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.172646 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.172646 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.172646 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.172646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 19da64d5-62e3-4b38-84f1-7d0971806ce9 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.172646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.172646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:27.172646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:27.172646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:27.172646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:27.172952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:43:27.172952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.172952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.172952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.172952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.172952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.172952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.172952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.172952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.172952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.172952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.172952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.172952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.173166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.173166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.173166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.173166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.173166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.173166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.173166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.173166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.173166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.173166 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 2655d772-0a40-4dd1-a462-c0b03b0b3b72 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:27.197389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-2655d772-0a40-4dd1-a462-c0b03b0b3b72, resource={'id': '2655d772-0a40-4dd1-a462-c0b03b0b3b72', 'name': 'tempest-VolumesBackupsTest-749072594-network', 'tenant_id': '0eb9bc78265643bc80f867d79e6a714a', 'project_id': '0eb9bc78265643bc80f867d79e6a714a', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['34e7cfb2-cc37-487a-addc-386e16fce028'], 'standard_attr_id': 1459, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:40:13Z', 'updated_at': '2026-06-01T11:40:14Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 61075}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.197730 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-2655d772-0a40-4dd1-a462-c0b03b0b3b72, col_values=(('external_ids', {'neutron:network_name': 'tempest-VolumesBackupsTest-749072594-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.198043 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-011496c2-d976-415f-b573-b1af2875bbae, external_ids_update=None, columns={'tag': 61075}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.198178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.199025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.199025 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.199025 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.199025 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.199025 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.199025 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.199025 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.199025 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.200175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 2655d772-0a40-4dd1-a462-c0b03b0b3b72 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.200175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.200175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:27.200175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:27.200175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:27.200175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:27.200175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:43:27.200175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.200175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.200175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.200175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.201374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.201374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.201374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.201374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.201374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.201374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.201374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.201374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.201374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.201374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.201374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.201374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.201374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.201374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.202294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.202294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.202294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.202294 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 3492f8e6-3ea9-42e6-986a-8ca849074669 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:27.227740 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-3492f8e6-3ea9-42e6-986a-8ca849074669, resource={'id': '3492f8e6-3ea9-42e6-986a-8ca849074669', 'name': 'tempest-TestEncryptedCinderVolumes-1394906684-network', 'tenant_id': 'd77c4e1763dc48c7a35074bdb31f0f9f', 'project_id': 'd77c4e1763dc48c7a35074bdb31f0f9f', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['ca56285f-442a-4f1a-8a7e-1b4f68f085a5'], 'standard_attr_id': 867, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:34:10Z', 'updated_at': '2026-06-01T11:34:10Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 23558}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.228146 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-3492f8e6-3ea9-42e6-986a-8ca849074669, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestEncryptedCinderVolumes-1394906684-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.228486 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-94d05f2e-959d-48f3-a414-21a304e82be7, external_ids_update=None, columns={'tag': 23558}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.228710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.230235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.230235 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.230235 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.230235 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.230235 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.230235 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.230235 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.230235 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.231305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 3492f8e6-3ea9-42e6-986a-8ca849074669 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.231305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.231305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:27.231305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:27.231305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:27.231305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:27.231305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:43:27.231305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.231305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.231305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.231305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.231943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.231943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.231943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.231943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.231943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.231943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.231943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.231943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.231943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.231943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.231943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.231943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.231943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.231943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.232669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.232669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.232669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.233030 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 4161094b-819d-4880-886d-6024ebdbe0cb (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:27.262317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-4161094b-819d-4880-886d-6024ebdbe0cb, resource={'id': '4161094b-819d-4880-886d-6024ebdbe0cb', 'name': 'tempest-TestVolumeBootPattern-213023248-network', 'tenant_id': 'd7b521081c864665a8943abfce50a25c', 'project_id': 'd7b521081c864665a8943abfce50a25c', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['86aa46bd-5f34-40c7-b4a1-34a2187eb1c4'], 'standard_attr_id': 1244, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:38:08Z', 'updated_at': '2026-06-01T11:38:09Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 21284}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.262747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-4161094b-819d-4880-886d-6024ebdbe0cb, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBootPattern-213023248-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.263053 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-e4bfbf35-4731-4003-838c-52b48b16ca81, external_ids_update=None, columns={'tag': 21284}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.263173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.263770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.263770 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.263770 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.263770 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.263770 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.263770 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.263770 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.263770 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.265208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 4161094b-819d-4880-886d-6024ebdbe0cb (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.265208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.265208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:27.265208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:27.265208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:27.265208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:27.265208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:43:27.265208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.265208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.265208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.265208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.265485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.265485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.265485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.265485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.265485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.265485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.265485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.265485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.265485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.265485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.265485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.265485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.265485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.265485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.265760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.265760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.265760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.265760 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 45586b37-8c68-4975-bc3b-8e3230ced693 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:27.292794 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-45586b37-8c68-4975-bc3b-8e3230ced693, resource={'id': '45586b37-8c68-4975-bc3b-8e3230ced693', 'name': 'tempest-TestEncryptedCinderVolumes-1366858945-network', 'tenant_id': '3fb7e4fd9c544b28a60ab0a49b43c46b', 'project_id': '3fb7e4fd9c544b28a60ab0a49b43c46b', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['ffbd1d55-ab6d-4849-8003-322763ee8a09'], 'standard_attr_id': 854, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:34:02Z', 'updated_at': '2026-06-01T11:34:03Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 1331}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.293150 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-45586b37-8c68-4975-bc3b-8e3230ced693, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestEncryptedCinderVolumes-1366858945-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.293463 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-64ca6c6b-97ac-45ef-842b-704b4e3b7689, external_ids_update=None, columns={'tag': 1331}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.293581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.294118 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.294118 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.294118 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.294118 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.294118 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.294118 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.294118 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.294118 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.295224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 45586b37-8c68-4975-bc3b-8e3230ced693 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.295224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.295224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:27.295224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:27.295224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:27.295224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:27.295224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:43:27.295224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.295224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.295224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.295224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.295573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.295573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.295573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.295573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.295573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.295573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.295573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.295573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.295573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.295573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.295573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.295573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.295573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.295573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.295907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.295907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.295907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.295907 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 95f21941-f51c-40a3-b07d-3588793d7c0e (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:27.328664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-95f21941-f51c-40a3-b07d-3588793d7c0e, resource={'id': '95f21941-f51c-40a3-b07d-3588793d7c0e', 'name': 'tempest-TestVolumeBootPattern-445281850-network', 'tenant_id': 'aa14eaa87f414863a6a7057001cc470c', 'project_id': 'aa14eaa87f414863a6a7057001cc470c', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['e9c91354-0006-49fe-9c88-46db6eb3be9e'], 'standard_attr_id': 1277, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:38:17Z', 'updated_at': '2026-06-01T11:38:18Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 43717}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.329064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-95f21941-f51c-40a3-b07d-3588793d7c0e, col_values=(('external_ids', {'neutron:network_name': 'tempest-TestVolumeBootPattern-445281850-network', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.329421 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-3b94e9a8-51b8-45d3-9554-6cd250f5a972, external_ids_update=None, columns={'tag': 43717}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.329611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.330276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.330276 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.330276 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.330276 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.330276 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.330276 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.330276 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.330276 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.331373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 95f21941-f51c-40a3-b07d-3588793d7c0e (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.331373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.331373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:27.331373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:27.331373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:27.331373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:27.331373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:43:27.331373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.331373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.331373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.331373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.331684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.331684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.331684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.331684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.331684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.331684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.331684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.331684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.331684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.331684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.331684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.331684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.331684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.331684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.331942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.331942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.331942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.331942 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource c6c73189-2fc5-412b-9ed5-038efd7341bc (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:27.362377 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, resource={'id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'name': 'public', 'tenant_id': '43b6be50ae544caeb6a73eced65fb6a7', 'project_id': '43b6be50ae544caeb6a73eced65fb6a7', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['21571d58-7621-44df-bdae-bad06600904d', '2bd944db-99d5-4573-8fc8-07e7942a1537'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-01T11:23:15Z', 'updated_at': '2026-06-01T11:23:22Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.362884 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.363350 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-4ecb0f57-2a36-41bc-88d1-5cf4f555b097, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.363626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.364311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.364311 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.364311 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.364311 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.364311 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.364311 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.364311 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.364311 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.365910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource c6c73189-2fc5-412b-9ed5-038efd7341bc (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.365910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.365910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:27.365910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:27.365910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:27.365910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:27.365910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:43:27.365910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.365910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.365910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.365910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.366152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.366152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.366152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.366152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.366152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.366152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.366152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.366152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.366152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.366152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.366152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.366152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.366152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.366152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.366431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.366431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.366431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.366431 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 2f364562-42b4-4604-88f0-e9200334f0f9 (type: security_groups) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:27.398339 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Successfully bumped revision number for resource 2f364562-42b4-4604-88f0-e9200334f0f9 (type: security_groups) to 3 Jun 01 11:43:27.398339 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 0887475e-f238-4c9c-ab00-0385551e6fa3 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:27.475314 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, resource={'id': '0887475e-f238-4c9c-ab00-0385551e6fa3', 'name': 'tempest-TestEncryptedCinderVolumes-1394906684-router', 'project_id': 'd77c4e1763dc48c7a35074bdb31f0f9f', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.211'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::221'}], 'enable_snat': True}, 'gw_port_id': '05e73b9a-1af7-4f42-b8ae-0d3d7873b442', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.211', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::221', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:34:11Z', 'updated_at': '2026-06-01T11:34:14Z', 'revision_number': 3, 'tenant_id': 'd77c4e1763dc48c7a35074bdb31f0f9f'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.475640 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, columns={'external_ids': {'neutron:router_name': 'tempest-TestEncryptedCinderVolumes-1394906684-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.475817 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "05e73b9a-1af7-4f42-b8ae-0d3d7873b442"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.475929 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "05e73b9a-1af7-4f42-b8ae-0d3d7873b442"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.476020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.476643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.476643 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.476643 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.476643 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.476643 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.476643 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.476643 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.476643 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.476865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router 0887475e-f238-4c9c-ab00-0385551e6fa3. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.481566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 0887475e-f238-4c9c-ab00-0385551e6fa3 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.481566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.481566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:27.481566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:27.481566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:27.481566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:27.481566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:43:27.481566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:43:27.481566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:43:27.481566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:43:27.481566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:43:27.481985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:43:27.481985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:43:27.481985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.481985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.481985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.481985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.481985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.481985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.481985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.481985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.481985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.481985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.481985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.481985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.482302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.482302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.482302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.482302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.482302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.482302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.482302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.482302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.482302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.482302 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 311f3cda-f2b3-45b2-81de-0fea8df583d6 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:27.548569 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, resource={'id': '311f3cda-f2b3-45b2-81de-0fea8df583d6', 'name': 'tempest-TestVolumeBootPattern-213023248-router', 'project_id': 'd7b521081c864665a8943abfce50a25c', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.56'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::3c'}], 'enable_snat': True}, 'gw_port_id': '9e0207ff-4677-4c9d-bc24-f5d5d6ca9f0d', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.56', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::3c', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:38:09Z', 'updated_at': '2026-06-01T11:38:14Z', 'revision_number': 3, 'tenant_id': 'd7b521081c864665a8943abfce50a25c'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.548778 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBootPattern-213023248-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.548778 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "9e0207ff-4677-4c9d-bc24-f5d5d6ca9f0d"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.548778 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "9e0207ff-4677-4c9d-bc24-f5d5d6ca9f0d"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.548778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.548778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.548778 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.548778 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.548778 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.549106 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.549106 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.549106 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.549106 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.549106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router 311f3cda-f2b3-45b2-81de-0fea8df583d6. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.549106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 311f3cda-f2b3-45b2-81de-0fea8df583d6 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.549106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.549106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:27.549106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:27.549106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:27.549106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:27.549428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:43:27.549428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:43:27.549428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:43:27.549428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:43:27.549428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:43:27.549428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:43:27.549428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:43:27.549428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.549428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.549428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.549428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.549428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.549428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.549692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.549692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.549692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.549692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.549692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.549692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.549692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.549692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.549692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.549692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.549692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.549692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.549692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.549914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.549914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.549914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 3e9267f2-318f-43d1-b28b-efabfd85699b (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:27.615931 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, resource={'id': '3e9267f2-318f-43d1-b28b-efabfd85699b', 'name': 'tempest-TestVolumeBootPattern-445281850-router', 'project_id': 'aa14eaa87f414863a6a7057001cc470c', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.50'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::7b'}], 'enable_snat': True}, 'gw_port_id': '4c6b3cf2-e705-4d91-a405-aea4ec4b1d91', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.50', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::7b', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:38:19Z', 'updated_at': '2026-06-01T11:38:21Z', 'revision_number': 3, 'tenant_id': 'aa14eaa87f414863a6a7057001cc470c'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.616267 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, columns={'external_ids': {'neutron:router_name': 'tempest-TestVolumeBootPattern-445281850-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.616494 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "4c6b3cf2-e705-4d91-a405-aea4ec4b1d91"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.616738 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "4c6b3cf2-e705-4d91-a405-aea4ec4b1d91"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.616912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.617433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.617433 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.617433 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.617433 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.617433 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.617433 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.617433 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.617433 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.617622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router 3e9267f2-318f-43d1-b28b-efabfd85699b. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.618958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 3e9267f2-318f-43d1-b28b-efabfd85699b (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.618958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.618958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:27.618958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:27.618958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:27.618958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:27.618958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:43:27.618958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:43:27.618958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:43:27.618958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:43:27.618958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:43:27.619254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:43:27.619254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:43:27.619254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.619254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.619254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.619254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.619254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.619254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.619254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.619254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.619254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.619254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.619254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.619254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.619518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.619518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.619518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.619518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.619518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.619518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.619518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.619518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.619518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.619518 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource a6b960b9-a555-46bb-bb6b-3e12a9085233 (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:27.691728 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, resource={'id': 'a6b960b9-a555-46bb-bb6b-3e12a9085233', 'name': 'tempest-VolumesBackupsTest-749072594-router', 'project_id': '0eb9bc78265643bc80f867d79e6a714a', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.85'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::202'}], 'enable_snat': True}, 'gw_port_id': '2cea6e44-6ac0-49de-97be-2553592819c9', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.85', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::202', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:40:14Z', 'updated_at': '2026-06-01T11:40:16Z', 'revision_number': 3, 'tenant_id': '0eb9bc78265643bc80f867d79e6a714a'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.692017 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, columns={'external_ids': {'neutron:router_name': 'tempest-VolumesBackupsTest-749072594-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.692235 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "2cea6e44-6ac0-49de-97be-2553592819c9"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.692365 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "2cea6e44-6ac0-49de-97be-2553592819c9"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.692494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.693016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.693016 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.693016 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.693016 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.693016 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.693016 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.693016 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.693016 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.693278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router a6b960b9-a555-46bb-bb6b-3e12a9085233. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.694769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource a6b960b9-a555-46bb-bb6b-3e12a9085233 (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.694769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.694769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:27.694769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:27.694769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:27.694769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:27.694769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:43:27.694769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:43:27.694769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:43:27.694769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:43:27.694769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:43:27.695102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:43:27.695102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:43:27.695102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.695102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.695102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.695102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.695102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.695102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.695102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.695102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.695102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.695102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.695102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.695102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.695379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.695379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.695379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.695379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.695379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.695379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.695379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.695379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.695379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.695379 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource d1c522ad-0f59-4939-9391-c3533f4cb09e (type: routers) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:27.763572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, resource={'id': 'd1c522ad-0f59-4939-9391-c3533f4cb09e', 'name': 'tempest-TestEncryptedCinderVolumes-1366858945-router', 'project_id': '3fb7e4fd9c544b28a60ab0a49b43c46b', 'admin_state_up': True, 'status': 'ACTIVE', 'enable_snat': True, 'external_gateway_info': {'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.221'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::189'}], 'enable_snat': True}, 'gw_port_id': '64c1bc42-615e-4bd7-b6c0-b6e62f628072', 'description': '', 'availability_zones': [], 'distributed': False, 'ha': True, 'ha_vr_id': 0, 'availability_zone_hints': [], 'enable_default_route_ecmp': False, 'enable_default_route_bfd': False, 'external_gateways': [{'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'external_fixed_ips': [{'ip_address': '172.24.5.221', 'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537'}, {'ip_address': '2001:db8::189', 'subnet_id': '21571d58-7621-44df-bdae-bad06600904d'}]}], 'routes': [], 'flavor_id': None, 'tags': [], 'created_at': '2026-06-01T11:34:04Z', 'updated_at': '2026-06-01T11:34:07Z', 'revision_number': 3, 'tenant_id': '3fb7e4fd9c544b28a60ab0a49b43c46b'}, resource_type=routers, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.763847 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterCommand(_result=None, name=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, columns={'external_ids': {'neutron:router_name': 'tempest-TestEncryptedCinderVolumes-1366858945-router', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': ''}, 'enabled': True}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.764073 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'from-lport'), ('priority', '=', 2002), ('match', '=', 'inport == "64c1bc42-615e-4bd7-b6c0-b6e62f628072"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.764207 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): QoSDelCommand(_result=None, switch=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, conditions=[('direction', '=', 'to-lport'), ('priority', '=', 2002), ('match', '=', 'outport == "64c1bc42-615e-4bd7-b6c0-b6e62f628072"')], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.764314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.764982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.764982 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.764982 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.764982 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.764982 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.764982 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.764982 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.764982 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.765188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_client [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Unable to update router d1c522ad-0f59-4939-9391-c3533f4cb09e. Error: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.767476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource d1c522ad-0f59-4939-9391-c3533f4cb09e (type: routers): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.767476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.767476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:27.767476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:27.767476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:27.767476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:27.767476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1637, in update_router Jun 01 11:43:27.767476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with excutils.save_and_reraise_exception(): Jun 01 11:43:27.767476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 266, in __exit__ Jun 01 11:43:27.767476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.force_reraise() Jun 01 11:43:27.767476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 232, in force_reraise Jun 01 11:43:27.767766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise self.value Jun 01 11:43:27.767766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1571, in update_router Jun 01 11:43:27.767766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.767766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.767766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.767766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.767766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.767766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.767766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.767766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.767766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.767766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.767766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.767766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.767999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.767999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.767999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.767999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.767999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.767999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.767999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.767999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.767999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.767999 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 05e73b9a-1af7-4f42-b8ae-0d3d7873b442 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:27.887138 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-05e73b9a-1af7-4f42-b8ae-0d3d7873b442, resource={'id': '05e73b9a-1af7-4f42-b8ae-0d3d7873b442', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:62:c5:5c', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '0887475e-f238-4c9c-ab00-0385551e6fa3', 'device_owner': 'network:router_gateway', 'standard_attr_id': 872, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.211'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::221'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:34:11Z', 'updated_at': '2026-06-01T11:34:14Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.887498 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-05e73b9a-1af7-4f42-b8ae-0d3d7873b442, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-0887475e-f238-4c9c-ab00-0385551e6fa3'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::221/64', '172.24.5.211/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.887864 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=05e73b9a-1af7-4f42-b8ae-0d3d7873b442, lrouter_port=lrp-05e73b9a-1af7-4f42-b8ae-0d3d7873b442, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:27.888112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.888816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:27.888816 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.888816 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:27.888816 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:27.888816 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.888816 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:27.888816 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.888816 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:27.889961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 05e73b9a-1af7-4f42-b8ae-0d3d7873b442 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.889961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:27.889961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:27.889961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:27.889961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:27.889961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:27.889961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:43:27.889961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:27.889961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.889961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.889961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:27.890331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:27.890331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:27.890331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:27.890331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:27.890331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:27.890331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:27.890331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:27.890331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:27.890331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:27.890331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:27.890331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:27.890331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:27.890331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:27.890331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:27.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:27.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:27.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:27.890685 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 2cea6e44-6ac0-49de-97be-2553592819c9 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:28.000989 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-2cea6e44-6ac0-49de-97be-2553592819c9, resource={'id': '2cea6e44-6ac0-49de-97be-2553592819c9', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:74:98:b6', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'a6b960b9-a555-46bb-bb6b-3e12a9085233', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1464, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.85'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::202'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:40:14Z', 'updated_at': '2026-06-01T11:40:19Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:28.001586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-2cea6e44-6ac0-49de-97be-2553592819c9, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.85/24', '2001:db8::202/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:28.001586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=2cea6e44-6ac0-49de-97be-2553592819c9, lrouter_port=lrp-2cea6e44-6ac0-49de-97be-2553592819c9, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:28.001586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:28.002035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:28.002035 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:28.002035 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:28.002035 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:28.002035 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:28.002035 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:28.002035 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:28.002035 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:28.002941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 2cea6e44-6ac0-49de-97be-2553592819c9 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:28.002941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:28.002941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:28.002941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:28.002941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:28.002941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:28.002941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:43:28.002941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:28.002941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:28.002941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:28.002941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:28.003238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:28.003238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:28.003238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:28.003238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:28.003238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:28.003238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:28.003238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:28.003238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:28.003238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:28.003238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:28.003238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:28.003238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:28.003238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:28.003238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:28.003604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:28.003604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:28.003604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:28.003604 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 4c6b3cf2-e705-4d91-a405-aea4ec4b1d91 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:28.120332 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-4c6b3cf2-e705-4d91-a405-aea4ec4b1d91, resource={'id': '4c6b3cf2-e705-4d91-a405-aea4ec4b1d91', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:93:83:e3', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '3e9267f2-318f-43d1-b28b-efabfd85699b', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1282, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.50'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::7b'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:38:19Z', 'updated_at': '2026-06-01T11:38:23Z', 'revision_number': 7, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:28.120510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-4c6b3cf2-e705-4d91-a405-aea4ec4b1d91, columns={'external_ids': {'neutron:revision_number': '7', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-3e9267f2-318f-43d1-b28b-efabfd85699b'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::7b/64', '172.24.5.50/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:28.120768 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=4c6b3cf2-e705-4d91-a405-aea4ec4b1d91, lrouter_port=lrp-4c6b3cf2-e705-4d91-a405-aea4ec4b1d91, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:28.120981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:28.121527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:28.121527 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:28.121527 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:28.121527 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:28.121527 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:28.121527 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:28.121527 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:28.121527 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:28.122347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 4c6b3cf2-e705-4d91-a405-aea4ec4b1d91 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:28.122347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:28.122347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:28.122347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:28.122347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:28.122347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:28.122347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:43:28.122347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:28.122347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:28.122347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:28.122347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:28.122631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:28.122631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:28.122631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:28.122631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:28.122631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:28.122631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:28.122631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:28.122631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:28.122631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:28.122631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:28.122631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:28.122631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:28.122631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:28.122631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:28.122874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:28.122874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:28.122874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:28.122874 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 64c1bc42-615e-4bd7-b6c0-b6e62f628072 (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:28.222713 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-64c1bc42-615e-4bd7-b6c0-b6e62f628072, resource={'id': '64c1bc42-615e-4bd7-b6c0-b6e62f628072', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:ec:e7:da', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'd1c522ad-0f59-4939-9391-c3533f4cb09e', 'device_owner': 'network:router_gateway', 'standard_attr_id': 860, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.221'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::189'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:34:04Z', 'updated_at': '2026-06-01T11:34:07Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:28.223127 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-64c1bc42-615e-4bd7-b6c0-b6e62f628072, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['2001:db8::189/64', '172.24.5.221/24'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:28.223396 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=64c1bc42-615e-4bd7-b6c0-b6e62f628072, lrouter_port=lrp-64c1bc42-615e-4bd7-b6c0-b6e62f628072, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:28.223650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:28.224258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:28.224258 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:28.224258 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:28.224258 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:28.224258 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:28.224258 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:28.224258 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:28.224258 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:28.225183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 64c1bc42-615e-4bd7-b6c0-b6e62f628072 (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:28.225183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:28.225183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:28.225183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:28.225183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:28.225183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:28.225183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:43:28.225183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:28.225183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:28.225183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:28.225183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:28.225462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:28.225462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:28.225462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:28.225462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:28.225462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:28.225462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:28.225462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:28.225462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:28.225462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:28.225462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:28.225462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:28.225462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:28.225462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:28.225462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:28.225724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:28.225724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:28.225724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:28.225724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 9e0207ff-4677-4c9d-bc24-f5d5d6ca9f0d (type: router_ports) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:43:28.330978 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=lrp-9e0207ff-4677-4c9d-bc24-f5d5d6ca9f0d, resource={'id': '9e0207ff-4677-4c9d-bc24-f5d5d6ca9f0d', 'name': '', 'network_id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'project_id': '', 'mac_address': 'fa:16:3e:cd:83:3c', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': '311f3cda-f2b3-45b2-81de-0fea8df583d6', 'device_owner': 'network:router_gateway', 'standard_attr_id': 1254, 'fixed_ips': [{'subnet_id': '2bd944db-99d5-4573-8fc8-07e7942a1537', 'ip_address': '172.24.5.56'}, {'subnet_id': '21571d58-7621-44df-bdae-bad06600904d', 'ip_address': '2001:db8::3c'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'np0000001841', 'binding:vif_type': 'ovs', 'binding:vif_details': {'port_filter': True, 'connectivity': 'l2', 'ovs_create_tap': False, 'bridge_name': 'br-int', 'datapath_type': 'system', 'bound_drivers': {'0': 'ovn'}}, 'port_security_enabled': False, 'tags': [], 'created_at': '2026-06-01T11:38:10Z', 'updated_at': '2026-06-01T11:38:14Z', 'revision_number': 5, 'tenant_id': ''}, resource_type=router_ports, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:28.331291 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): UpdateLRouterPortCommand(_result=None, name=lrp-9e0207ff-4677-4c9d-bc24-f5d5d6ca9f0d, columns={'external_ids': {'neutron:revision_number': '5', 'neutron:subnet_ids': '2bd944db-99d5-4573-8fc8-07e7942a1537 21571d58-7621-44df-bdae-bad06600904d', 'neutron:network_name': 'neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc', 'neutron:is_ext_gw': 'True', 'neutron:router_name': 'neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6'}, 'options': {'reside-on-redirect-chassis': 'true', 'gateway_mtu': '1372'}, 'networks': ['172.24.5.56/24', '2001:db8::3c/64'], 'ipv6_ra_configs': {}}, if_exists=False) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:28.331615 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLRouterPortInLSwitchPortCommand(_result=None, lswitch_port=9e0207ff-4677-4c9d-bc24-f5d5d6ca9f0d, lrouter_port=lrp-9e0207ff-4677-4c9d-bc24-f5d5d6ca9f0d, is_gw_port=True, if_exists=True, lsp_address=router) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:28.331816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:28.332391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:28.332391 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:28.332391 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:28.332391 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:28.332391 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:28.332391 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:28.332391 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:28.332391 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:28.333233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 9e0207ff-4677-4c9d-bc24-f5d5d6ca9f0d (type: router_ports): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:28.333233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:28.333233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:43:28.333233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:43:28.333233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:43:28.333233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:43:28.333233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 1978, in update_router_port Jun 01 11:43:28.333233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:28.333233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:28.333233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:28.333233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:28.333500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:28.333500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:28.333500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:28.333500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:28.333500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:28.333500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:28.333500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:28.333500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:28.333500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:28.333500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:28.333500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:28.333500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:28.333500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:28.333500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:28.333868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:28.333868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:28.333868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:28.333868 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 0d8a7743-c388-4bbb-ae7d-1969d3485e98 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:43:28.333868 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(0d8a7743-c388-4bbb-ae7d-1969d3485e98) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:43:28.339123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 28bc6791-98f8-4481-b4b2-17017518d801 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:43:28.339353 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(28bc6791-98f8-4481-b4b2-17017518d801) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:43:28.343730 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 2d16a78e-ca9f-424a-b9f7-ced2bf3c6322 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:43:28.343968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(2d16a78e-ca9f-424a-b9f7-ced2bf3c6322) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:43:28.349910 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 51ee1662-ddb6-47da-a1b7-1e9ce15ff013 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:43:28.350158 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(51ee1662-ddb6-47da-a1b7-1e9ce15ff013) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:43:28.356675 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 5d749c57-2ae1-4fda-be9f-e56719fb44b6 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:43:28.356948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(5d749c57-2ae1-4fda-be9f-e56719fb44b6) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:43:28.362236 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 5fe3a0af-f68b-4cc7-947f-93f6132ea669 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:43:28.362395 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(5fe3a0af-f68b-4cc7-947f-93f6132ea669) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:43:28.371448 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 80f365b5-64f5-408d-9344-ac938776f490 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:43:28.371663 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(80f365b5-64f5-408d-9344-ac938776f490) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:43:28.379166 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 91ac5e2c-0c45-47b0-a854-6f34ff4755d1 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:43:28.379427 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(91ac5e2c-0c45-47b0-a854-6f34ff4755d1) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:43:28.384393 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 967fd2d1-7e83-4ba3-b4dc-52b9d5aba4fc (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:43:28.384709 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(967fd2d1-7e83-4ba3-b4dc-52b9d5aba4fc) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:43:28.390168 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 9b73ca6c-f685-4281-ba60-7efca4960ac0 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:43:28.390515 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(9b73ca6c-f685-4281-ba60-7efca4960ac0) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:43:28.396238 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource a6420187-c461-4cb3-93e3-5b4ec7b21a5d (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:43:28.396426 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(a6420187-c461-4cb3-93e3-5b4ec7b21a5d) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:43:28.400643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource b76e8c43-e6a5-4e0e-ae38-a143937e6670 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:43:28.400834 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(b76e8c43-e6a5-4e0e-ae38-a143937e6670) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:43:28.406349 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource f22be993-3abd-472b-8eba-a874c4258a6f (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:43:28.406633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(f22be993-3abd-472b-8eba-a874c4258a6f) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:43:28.411155 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_for_inconsistencies finished in 1.318 seconds Jun 01 11:43:31.917135 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:31.917719 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:31.917719 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:31.918286 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:31.918588 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:31.918747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:31.918881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:31.918997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:31.919123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:31.919244 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:31.919383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:31.919932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:31.919932 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:31.919932 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:31.919932 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:31.919932 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:31.919932 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:31.919932 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:31.919932 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:31.920778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:31.920778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:31.920778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:31.920778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:31.920778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:31.920778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:31.920778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:31.920778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:31.920778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:31.920778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:31.920778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:31.920778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:31.921180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:31.921180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:31.921180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:31.921180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:31.921180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:31.921180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:31.921180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:31.921180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:31.921180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:31.921180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:31.921180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:31.921180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:31.921180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:31.921630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:31.921630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:31.921917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:31.921917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:31.921917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:31.921917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:31.921917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:31.921917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:31.921917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:31.921917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:31.921917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:31.921917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:31.921917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:31.921917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:31.921917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:31.921917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:31.921917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:31.921917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:31.922303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:31.922711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:31.922711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:31.922711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:31.988049 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:31.988216 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:31.988306 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:31.988396 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:43:31.988767 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:31.988968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:31.989495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:31.989495 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:31.989495 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:31.989495 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:31.989495 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:31.989495 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:31.989495 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:31.989495 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:31.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:31.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:31.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:31.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:31.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:31.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:31.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:31.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:31.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:31.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:31.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:31.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:31.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:31.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:31.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:31.990472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:31.990472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:31.990472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:31.990472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:31.990472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:31.990472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:31.990472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:31.990472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:31.990472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:31.990772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:31.990772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:31.990772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:31.990772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:31.990772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:31.990772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:31.990772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:31.990772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:43:31.990772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:31.990772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:31.990772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:31.990772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:31.991114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:31.991114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:31.991114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:31.991114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:31.991114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:31.991114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:31.991114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:31.991114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:31.991114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:31.991114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:31.991114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:31.991114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:31.991114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:31.991114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:31.991518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:31.991518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:31.991518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:31.991518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:31.991518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:31.991518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:31.991518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:31.991518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:31.991518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:31.991518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:31.991518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:31.991518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:31.991518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:31.991518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:31.991973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:31.992573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:31.992573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:31.992573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:31.992573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:31.992573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:31.992573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:31.992573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:31.992573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:31.992573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:31.992573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:31.992573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:31.992573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:32.004816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:32.005137 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:32.005247 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:32.005369 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:32.005788 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:32.006004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.006434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:32.006434 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:32.006434 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:32.006434 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:32.006434 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:32.006434 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:32.006434 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.006434 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:32.007021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.007021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:32.007021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:32.007021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:32.007021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:32.007021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:32.007021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:32.007021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:32.007021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:32.007021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:32.007021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:32.007021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:32.007021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:32.007021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:32.007021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:32.007291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:32.007291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:32.007291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:32.007291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:32.007291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:32.007291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:32.007291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:32.007291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.007291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:32.007676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.007676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:32.007676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:32.007676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:32.007676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:32.007676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:32.007676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:32.007676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:32.007676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:32.007676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:32.007676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:32.007899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:32.007899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:32.007899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:32.007899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:32.007899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:32.007899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:32.007899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:32.007899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:32.007899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:32.007899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:32.007899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:32.007899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:32.007899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:32.007899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:32.008128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:32.008128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:32.008128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.008128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:32.008407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:32.008752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:32.009050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:32.009050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:32.009050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.009050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:32.033011 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:32.033362 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:32.033490 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:32.033713 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:32.034124 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:32.034396 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:32.034717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:32.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:32.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:32.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:32.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:32.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:32.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:32.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:32.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:32.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:32.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:32.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:32.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:32.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:32.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:32.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:32.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:32.036094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:32.036498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:32.036498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:32.036498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:32.036498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:32.036498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:32.036498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:32.036498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:32.036498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:32.036498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:32.036498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:32.036498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:32.036498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:32.036498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:32.036934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.036934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:32.037175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.037175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:32.037175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:32.037175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:32.037175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:32.037175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:32.037175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:32.037175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:32.037175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:32.037175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:32.037175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:32.037175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:32.037175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:32.037175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:32.037175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:32.037175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:32.037549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:32.037882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:32.037882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.037882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:32.057875 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:32.058244 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:32.058390 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:32.059132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:32.092288 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:32.092595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.093284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:32.093284 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:32.093284 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:32.093284 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:32.093284 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:32.093284 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:32.093284 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.093284 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:32.094395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.094395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:32.094395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:32.094395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:32.094395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:32.094395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:32.094395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:32.094395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:32.094395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:32.094395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:32.094395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:32.094395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:32.094720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:32.094720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:32.094720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:32.094720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:32.094720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:32.094720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:32.094720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:32.094720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:32.094720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:32.094720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:32.094720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:32.094720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:32.094720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:32.094964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.094964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:32.095418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.095418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:32.095418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:32.095418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:32.095418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:32.095418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:32.095418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:32.095418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:32.095418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:32.095418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:32.095418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:32.095418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:32.095418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:32.095418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:32.095418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:32.095418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:32.095770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:32.096094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:32.096094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:32.096094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:36.917497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:36.917996 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:36.917996 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:36.918639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:36.918796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:36.918933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:36.919073 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:36.919201 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:36.919332 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:36.919475 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:36.920289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:36.920289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:36.920289 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:36.920289 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:36.920289 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:36.920289 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:36.920289 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:36.920289 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:36.920289 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:36.920914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:36.920914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:36.920914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:36.920914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:36.920914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:36.920914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:36.920914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:36.920914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:36.920914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:36.920914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:36.920914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:36.920914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:36.922057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:36.922057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:36.922057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:36.922057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:36.922057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:36.922057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:36.922057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:36.922057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:36.922057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:36.922057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:36.922057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:36.922057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:36.922057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:36.922411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:36.922411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:36.922411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:36.922411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:36.922411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:36.922411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:36.922411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:36.922411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:36.922411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:36.922411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:36.922411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:36.922411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:36.922411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:36.922411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:36.922411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:36.922851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:36.923361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:36.923361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:36.923361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:36.923361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:36.923361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:36.923361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:36.988455 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:36.988621 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:36.988705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:36.988788 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:43:36.989075 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:36.989231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:36.989620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:36.989620 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:36.989620 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:36.989620 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:36.989620 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:36.989620 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:36.989620 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:36.989620 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:36.990107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:36.990107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:36.990107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:36.990107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:36.990107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:36.990107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:36.990107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:36.990107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:36.990107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:36.990107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:36.990107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:36.990107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:36.990107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:36.990107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:36.990107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:36.990425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:36.990425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:36.990425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:36.990425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:36.990425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:36.990425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:36.990425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:36.990425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:36.990425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:36.990680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:36.990680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:36.990680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:36.990680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:36.990680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:36.990680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:36.990680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:36.990680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:43:36.990680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:36.990680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:36.990680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:36.990680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:36.990910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:36.990910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:36.990910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:36.990910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:36.990910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:36.990910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:36.990910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:36.990910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:36.990910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:36.990910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:36.990910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:36.990910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:36.990910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:36.990910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:36.991180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:36.991180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:36.991180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:36.991180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:36.991331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:36.991331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:36.991331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:36.991331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:36.991331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:36.991331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:36.991331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:36.991331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:36.991331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:36.991331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:36.991331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:36.991331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:36.991331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:36.991331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:36.991331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:36.991331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:36.991662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:36.991992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:36.991992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:36.991992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:36.991992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:36.991992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:36.991992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:36.991992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:37.005278 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:37.005395 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:37.005483 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:37.005596 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:37.005862 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:37.006039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.006461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:37.006461 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:37.006461 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:37.006461 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:37.006461 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:37.006461 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:37.006461 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.006461 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:37.006961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.006961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:37.006961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:37.006961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:37.006961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:37.006961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:37.006961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:37.006961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:37.006961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:37.006961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:37.006961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:37.006961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:37.006961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:37.006961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:37.006961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:37.007247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:37.007247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:37.007247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:37.007247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:37.007247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:37.007247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:37.007247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:37.007247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.007247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:37.007642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.007642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:37.007642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:37.007642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:37.007642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:37.007642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:37.007642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:37.007642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:37.007642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:37.007642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:37.007642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:37.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:37.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:37.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:37.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:37.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:37.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:37.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:37.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:37.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:37.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:37.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:37.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:37.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:37.007868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:37.008110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:37.008110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:37.008110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.008110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:37.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:37.008725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:37.009057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:37.009057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:37.009057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.009057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:37.033361 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:37.033603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:37.033687 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:37.033815 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:37.034048 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:37.034215 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:37.034418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:37.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:37.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:37.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:37.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:37.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:37.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.035095 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:37.036302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.036302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:37.036302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:37.036302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:37.036302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:37.036302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:37.036302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:37.036302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:37.036302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:37.036302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:37.036302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:37.036302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:37.036689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:37.036689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:37.036689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:37.036689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:37.036689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:37.036689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:37.036689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:37.036689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:37.036689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:37.036689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:37.036689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:37.036689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:37.036689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:37.036953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.036953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:37.037421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.037421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:37.037421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:37.037421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:37.037421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:37.037421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:37.037421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:37.037421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:37.037421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:37.037421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:37.037421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:37.037421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:37.037421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:37.037421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:37.037421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:37.037421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:37.037899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:37.038320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:37.038320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.038320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:37.058367 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:37.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:37.059269 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:37.059997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:37.099997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:37.100235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.101319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:37.101319 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:37.101319 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:37.101319 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:37.101319 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:37.101319 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:37.101319 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.101319 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:37.102165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.102165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:37.102165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:37.102165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:37.102165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:37.102165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:37.102165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:37.102165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:37.102165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:37.102165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:37.102165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:37.102165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:37.102578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:37.102578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:37.102578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:37.102578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:37.102578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:37.102578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:37.102578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:37.102578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:37.102578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:37.102578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:37.102578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:37.102578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:37.102578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:37.102982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.102982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:37.103670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.103670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:37.103670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:37.103670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:37.103670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:37.103670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:37.103670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:37.103670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:37.103670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:37.103670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:37.103670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:37.103670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:37.103670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:37.103670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:37.103670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:37.103670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:37.104033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:37.104350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:37.104350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:37.104350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:41.918102 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:41.918953 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:41.919105 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:41.920233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:41.920476 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:41.920672 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:41.920850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:41.921015 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:41.921173 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:41.921318 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:41.921494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:41.922152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:41.922152 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:41.922152 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:41.922152 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:41.922152 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:41.922152 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:41.922152 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:41.922152 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:41.922935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:41.922935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:41.922935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:41.922935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:41.922935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:41.922935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:41.922935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:41.922935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:41.922935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:41.922935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:41.922935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:41.922935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:41.923453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:41.923453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:41.923453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:41.923453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:41.923453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:41.923453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:41.923453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:41.923453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:41.923453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:41.923453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:41.923453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:41.923453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:41.923453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:41.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:41.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:41.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:41.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:41.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:41.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:41.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:41.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:41.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:41.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:41.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:41.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:41.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:41.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:41.923976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:41.926804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:41.927192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:41.927192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:41.927192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:41.927192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:41.927192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:41.927192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:41.988877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:41.989238 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:41.989334 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:41.989437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:43:41.989814 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:41.989966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:41.990572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:41.990572 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:41.990572 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:41.990572 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:41.990572 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:41.990572 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:41.990572 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:41.990572 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:41.991089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:41.991089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:41.991089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:41.991089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:41.991089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:41.991089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:41.991089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:41.991089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:41.991089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:41.991089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:41.991089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:41.991089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:41.991089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:41.991089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:41.991089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:41.991497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:41.991497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:41.991497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:41.991497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:41.991497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:41.991497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:41.991497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:41.991497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:41.991497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:41.991737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:41.991737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:41.991737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:41.991737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:41.991737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:41.991737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:41.991737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:41.991737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:43:41.991737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:41.991737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:41.991737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:41.991737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:41.991989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:41.991989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:41.991989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:41.991989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:41.991989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:41.991989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:41.991989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:41.991989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:41.991989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:41.991989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:41.991989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:41.991989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:41.991989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:41.991989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:41.992220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:41.992220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:41.992220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:41.992220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:41.992391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:41.992391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:41.992391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:41.992391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:41.992391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:41.992391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:41.992391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:41.992391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:41.992391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:41.992391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:41.992391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:41.992391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:41.992391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:41.992391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:41.992391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:41.992391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:41.992669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:41.992973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:41.992973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:41.992973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:41.992973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:41.992973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:41.992973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:41.992973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:42.005703 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:42.005921 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:42.006015 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:42.006136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:42.006479 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:42.006728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.007176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:42.007176 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:42.007176 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:42.007176 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:42.007176 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:42.007176 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:42.007176 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.007176 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:42.007843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.007843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:42.007843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:42.007843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:42.007843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:42.007843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:42.007843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:42.007843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:42.007843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:42.007843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:42.007843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:42.007843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:42.007843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:42.007843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:42.007843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:42.008154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:42.008154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:42.008154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:42.008154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:42.008154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:42.008154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:42.008154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:42.008154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.008154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:42.008561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.008561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:42.008561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:42.008561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:42.008561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:42.008561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:42.008561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:42.008561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:42.008561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:42.008561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:42.008561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:42.008776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:42.008776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:42.008776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:42.008776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:42.008776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:42.008776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:42.008776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:42.008776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:42.008776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:42.008776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:42.008776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:42.008776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:42.008776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:42.008776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:42.009004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:42.009004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:42.009004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.009004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:42.009418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:42.009733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:42.010028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:42.010028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:42.010028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.010028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:42.033757 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:42.034045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:42.034142 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:42.034295 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:42.034619 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:42.034833 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:42.035077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:42.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:42.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:42.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:42.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:42.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:42.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:42.036501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.036501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:42.036501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:42.036501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:42.036501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:42.036501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:42.036501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:42.036501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:42.036501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:42.036501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:42.036501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:42.036501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:42.036762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:42.036762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:42.036762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:42.036762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:42.036762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:42.036762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:42.036762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:42.036762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:42.036762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:42.036762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:42.036762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:42.036762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:42.036762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:42.036994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.036994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:42.037646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.037646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:42.037646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:42.037646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:42.037646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:42.037646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:42.037646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:42.037646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:42.037646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:42.037646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:42.037646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:42.037646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:42.037646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:42.037646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:42.037646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:42.037646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:42.037930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:42.038236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:42.038236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.038236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:42.059340 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:42.059686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:42.059820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:42.060681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:42.090874 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:42.091012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.091451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:42.091451 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:42.091451 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:42.091451 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:42.091451 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:42.091451 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:42.091451 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.091451 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:42.092144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.092144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:42.092144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:42.092144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:42.092144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:42.092144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:42.092144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:42.092144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:42.092144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:42.092144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:42.092144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:42.092144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:42.092528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:42.092528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:42.092528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:42.092528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:42.092528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:42.092528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:42.092528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:42.092528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:42.092528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:42.092528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:42.092528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:42.092528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:42.092528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:42.092926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.092926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:42.092926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.092926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:42.092926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:42.092926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:42.092926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:42.092926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:42.092926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:42.092926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:42.092926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:42.092926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:42.092926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:42.092926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:42.092926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:42.093335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:42.093894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:42.093894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:42.093894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:42.093894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:42.093894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:42.093894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:46.919003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:46.920236 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:46.920236 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:46.920975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:46.921618 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-0887475e-f238-4c9c-ab00-0385551e6fa3, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:46.921618 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:46.921618 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:46.921618 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:46.921618 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:46.921881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-d1c522ad-0f59-4939-9391-c3533f4cb09e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:46.921881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:46.922233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:46.922233 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:46.922233 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:46.922233 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:46.922233 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:46.922233 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:46.922233 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:46.922233 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:46.923132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:46.923132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:46.923132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:46.923132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:46.923132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:46.923132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:46.923132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:46.923132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:46.923132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:46.923132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:46.923132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:46.923132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:46.923387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:46.923387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:46.923387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:46.923387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:46.923387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:46.923387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:46.923387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:46.923387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:46.923387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:46.923387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:46.923387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:46.923387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:46.923387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:46.923667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:46.923667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:46.924046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:46.924046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:46.924046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:46.924046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:46.924046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:46.924046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:46.924046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:46.924046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:46.924046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:46.924046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:46.924046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:46.924046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:46.924046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:46.924046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:46.924046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:46.924046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:46.924323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:46.924663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:46.924663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:46.924663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:46.989382 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:46.989997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:46.989997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:46.989997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:43:46.990927 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:46.990927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:46.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:46.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:46.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:46.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:46.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:46.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:46.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:46.992172 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:46.992424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:46.992424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:46.992424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:46.992424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:46.992424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:46.992424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:46.992424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:46.992424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:46.992424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:46.992424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:46.992424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:46.992424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:46.992424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:46.992424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:46.992424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:46.992798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:46.992798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:46.992798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:46.992798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:46.992798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:46.992798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:46.992798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:46.992798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:46.992798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:46.993171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:46.993171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:46.993171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:46.993171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:46.993171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:46.993171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:46.993171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:46.993171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:43:46.993171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:46.993171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:46.993171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:46.993171 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:46.993505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:46.993505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:46.993505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:46.993505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:46.993505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:46.993505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:46.993505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:46.993505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:46.993505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:46.993505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:46.993505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:46.993505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:46.993505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:46.993505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:46.993850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:46.993850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:46.993850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:46.993850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:46.993970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:46.993970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:46.993970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:46.993970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:46.993970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:46.993970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:46.993970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:46.993970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:46.993970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:46.993970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:46.993970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:46.993970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:46.993970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:46.993970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:46.993970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:46.993970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:46.994366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:46.994966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:46.994966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:46.994966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:46.994966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:46.994966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:46.994966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:46.994966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:47.006324 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:47.006683 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:47.006783 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:47.006933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:47.007575 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:47.008274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.008916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:47.008916 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:47.008916 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:47.008916 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:47.008916 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:47.008916 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:47.008916 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.008916 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:47.009803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.009803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:47.009803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:47.009803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:47.009803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:47.009803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:47.009803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:47.009803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:47.009803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:47.009803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:47.009803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:47.009803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:47.009803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:47.009803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:47.009803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:47.010112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:47.010112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:47.010112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:47.010112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:47.010112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:47.010112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:47.010112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:47.010112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.010112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:47.010672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.010672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:47.010672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:47.010672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:47.010672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:47.010672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:47.010672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:47.010672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:47.010672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:47.010672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:47.010672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:47.010978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:47.010978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:47.010978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:47.010978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:47.010978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:47.010978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:47.010978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:47.010978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:47.010978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:47.010978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:47.010978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:47.010978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:47.010978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:47.010978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:47.011336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:47.011336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:47.011336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.011336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:47.011827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:47.012230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:47.012704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:47.012704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:47.012704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.012704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:47.034182 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:47.035126 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:47.035266 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:47.035454 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:47.035951 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:47.036269 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:47.036955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.037564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:47.037564 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:47.037564 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:47.037564 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:47.037564 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:47.037564 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:47.037564 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.037564 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:47.038807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.038807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:47.038807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:47.038807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:47.038807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:47.038807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:47.038807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:47.038807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:47.038807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:47.038807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:47.038807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:47.038807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:47.039106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:47.039106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:47.039106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:47.039106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:47.039106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:47.039106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:47.039106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:47.039106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:47.039106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:47.039106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:47.039106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:47.039106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:47.039106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:47.039391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.039391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:47.040089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.040089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:47.040089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:47.040089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:47.040089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:47.040089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:47.040089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:47.040089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:47.040089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:47.040089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:47.040089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:47.040089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:47.040089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:47.040089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:47.040089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:47.040089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:47.040492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:47.040845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:47.040845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.040845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:47.059752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:47.060086 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:47.060232 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:47.060963 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:47.098895 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:47.099149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.099780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:47.099780 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:47.099780 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:47.099780 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:47.099780 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:47.099780 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:47.099780 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.099780 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:47.100711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.100711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:47.100711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:47.100711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:47.100711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:47.100711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:47.100711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:47.100711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:47.100711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:47.100711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:47.100711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:47.100711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:47.101099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:47.101099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:47.101099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:47.101099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:47.101099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:47.101099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:47.101099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:47.101099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:47.101099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:47.101099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:47.101099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:47.101099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:47.101099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:47.101486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.101486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:47.101712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.101712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:47.101712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:47.101712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:47.101712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:47.101712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:47.101712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:47.101712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:47.101712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:47.101712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:47.101712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:47.101712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:47.101712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:47.101712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:47.101712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:47.101712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:47.102260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:47.102843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:47.102843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:47.102843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:51.920049 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:51.920979 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:51.920979 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:51.921684 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:51.921855 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:51.921984 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:51.922131 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:51.922287 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:51.922427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:51.922911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:51.922911 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:51.922911 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:51.922911 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:51.922911 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:51.922911 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:51.922911 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:51.922911 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:51.923670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:51.923670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:51.923670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:51.923670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:51.923670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:51.923670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:51.923670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:51.923670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:51.923670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:51.923670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:51.923670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:51.923670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:51.924130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:51.924130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:51.924130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:51.924130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:51.924130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:51.924130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:51.924130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:51.924130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:51.924130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:51.924130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:51.924130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:51.924130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:51.924130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:51.924536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:51.924536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:51.924536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:51.924536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:51.924536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:51.924536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:51.924536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:51.924536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:51.924536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:51.924536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:51.924536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:51.924536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:51.924536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:51.924536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:51.924536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:51.924898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:51.925284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:51.925284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:51.925284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:51.925284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:51.925284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:51.925284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:51.989815 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:51.990195 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:51.990351 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:51.990528 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:43:51.991441 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:51.991441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:51.991536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:51.991536 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:51.991536 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:51.991536 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:51.991536 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:51.991536 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:51.991536 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:51.991536 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:51.992243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:51.992243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:51.992243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:51.992243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:51.992243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:51.992243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:51.992243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:51.992243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:51.992243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:51.992243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:51.992243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:51.992243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:51.992243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:51.992243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:51.992243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:51.993085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:51.993085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:51.993085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:51.993085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:51.993085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:51.993085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:51.993085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:51.993085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:51.993085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:51.993364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:51.993364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:51.993364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:51.993364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:51.993364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:51.993364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:51.993364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:51.993364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:43:51.993364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:51.993364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:51.993364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:51.993364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:51.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:51.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:51.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:51.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:51.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:51.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:51.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:51.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:51.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:51.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:51.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:51.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:51.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:51.993705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:51.994112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:51.994112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:51.994112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:51.994112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:51.994363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:51.994363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:51.994363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:51.994363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:51.994363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:51.994363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:51.994363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:51.994363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:51.994363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:51.994363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:51.994363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:51.994363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:51.994363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:51.994363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:51.994363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:51.994363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:51.994737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:51.995009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:51.995009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:51.995009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:51.995009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:51.995009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:51.995009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:51.995009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:52.006709 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:52.007050 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:52.007178 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:52.007307 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:52.007679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:52.008249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.008374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:52.008374 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:52.008374 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:52.008374 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:52.008374 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:52.008374 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:52.008374 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.008374 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:52.008953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.008953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:52.008953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:52.008953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:52.008953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:52.008953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:52.008953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:52.008953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:52.008953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:52.008953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:52.008953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:52.008953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:52.008953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:52.008953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:52.008953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:52.009258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:52.009258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:52.009258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:52.009258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:52.009258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:52.009258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:52.009258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:52.009258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.009258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:52.009554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.009554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:52.009554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:52.009554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:52.009554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:52.009554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:52.009554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:52.009554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:52.009554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:52.009554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:52.009554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:52.009815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:52.009815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:52.009815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:52.009815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:52.009815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:52.009815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:52.009815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:52.009815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:52.009815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:52.009815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:52.009815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:52.009815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:52.009815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:52.009815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:52.010035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:52.010035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:52.010035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.010035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:52.010245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:52.010598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:52.010970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:52.010970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:52.010970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.010970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:52.034999 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:52.035475 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:52.035475 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:52.035650 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:52.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:52.036213 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:52.036426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.036953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:52.036953 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:52.036953 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:52.036953 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:52.036953 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:52.036953 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:52.036953 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.036953 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:52.037717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.037717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:52.037717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:52.037717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:52.037717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:52.037717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:52.037717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:52.037717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:52.037717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:52.037717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:52.037717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:52.037717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:52.038067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:52.038067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:52.038067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:52.038067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:52.038067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:52.038067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:52.038067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:52.038067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:52.038067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:52.038067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:52.038067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:52.038067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:52.038067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:52.038422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.038422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:52.038729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.038729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:52.038729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:52.038729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:52.038729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:52.038729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:52.038729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:52.038729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:52.038729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:52.038729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:52.038729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:52.038729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:52.038729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:52.038729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:52.038729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:52.038729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:52.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:52.039468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:52.039468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.039468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:52.060160 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:52.060432 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:52.060609 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:52.061427 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:52.098193 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:52.098193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.098394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:52.098394 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:52.098394 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:52.098394 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:52.098394 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:52.098394 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:52.098394 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.098394 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:52.099105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.099105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:52.099105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:52.099105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:52.099105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:52.099105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:52.099105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:52.099105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:52.099105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:52.099105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:52.099105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:52.099105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:52.099764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:52.099764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:52.099764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:52.099764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:52.099764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:52.099764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:52.099764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:52.099764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:52.099764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:52.099764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:52.099764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:52.099764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:52.099764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:52.100210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.100210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:52.100210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.100210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:52.100210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:52.100210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:52.100210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:52.100210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:52.100210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:52.100210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:52.100210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:52.100210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:52.100210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:52.100210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:52.100210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:52.101265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:52.101727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:52.101727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:52.101727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:52.101727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:52.101727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:52.101727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:56.920475 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:56.921649 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:56.921752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:56.922499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:56.922742 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:56.922931 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:56.923064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:56.923222 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:56.923366 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:56.923515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:56.924005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:56.924005 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:56.924005 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:56.924005 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:56.924005 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:56.924005 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:56.924005 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:56.924005 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:56.924783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:56.924783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:56.924783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:56.924783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:56.924783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:56.924783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:56.924783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:56.924783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:56.924783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:56.924783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:56.924783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:56.924783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:56.925152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:56.925152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:56.925152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:56.925152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:56.925152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:56.925152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:56.925152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:56.925152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:56.925152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:56.925152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:56.925152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:56.925152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:56.925152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:56.925521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:56.925521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:56.925651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:56.925651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:56.925651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:56.925651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:56.925651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:56.925651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:56.925651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:56.925651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:56.925651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:56.925651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:56.925651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:56.925651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:56.925651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:56.925651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:56.925651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:56.925651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:56.926119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:56.926605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:56.926605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:56.926605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:56.990212 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:56.991093 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:56.991158 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:56.991267 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:43:56.991926 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:56.992115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:56.992664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:56.992664 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:56.992664 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:56.992664 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:56.992664 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:56.992664 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:56.992664 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:56.992664 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:56.993286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:56.993286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:56.993286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:56.993286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:56.993286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:56.993286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:56.993286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:56.993286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:56.993286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:56.993286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:56.993286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:56.993286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:56.993286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:56.993286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:56.993286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:56.993744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:56.993744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:56.993744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:56.993744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:56.993744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:56.993744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:56.993744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:56.993744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:56.993744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:56.994258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:56.994258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:56.994258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:56.994258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:56.994258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:56.994258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:56.994258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:56.994258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:43:56.994258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:56.994258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:56.994258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:56.994258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:56.994646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:56.994646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:56.994646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:56.994646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:56.994646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:56.994646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:56.994646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:56.994646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:56.994646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:56.994646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:56.994646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:56.994646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:56.994646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:56.994646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:56.995249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:56.995249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:56.995249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:56.995249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:56.995249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:56.995249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:56.995249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:56.995249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:56.995249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:56.995249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:56.995249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:56.995249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:56.995249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:56.995249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:56.995858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:56.996373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:56.996373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:56.996373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:56.996373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:56.996373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:56.996373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:56.996373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:56.996373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:56.996373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:56.996373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:56.996373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:56.996373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:57.007092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:57.007638 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:57.007752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:57.007859 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:57.008204 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:57.008427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.009269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:57.009269 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:57.009269 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:57.009269 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:57.009269 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:57.009269 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:57.009269 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.009269 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:57.009906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.009906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:43:57.009906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:57.009906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:57.009906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:57.009906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:57.009906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:57.009906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:43:57.009906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:57.009906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:43:57.009906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:57.009906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:43:57.009906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:57.009906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:43:57.009906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:43:57.010518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:57.010518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:43:57.010518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:57.010518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:43:57.010518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:43:57.010518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:57.010518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:43:57.010518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.010518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:43:57.010518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.010518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:57.010518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:57.010518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:57.010518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:57.010940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:57.010940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:57.010940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:57.010940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:57.010940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:57.010940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:57.010940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:57.010940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:57.010940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:57.010940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:57.010940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:57.010940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:57.010940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:57.011330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:57.011330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:57.011330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:57.011330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:57.011330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:57.011330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:57.011330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:57.011330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:57.011330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:57.011330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.011330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:57.011330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:43:57.011716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.012245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:57.035374 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:57.037489 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:57.038252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:57.038252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:57.038252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:57.038693 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:57.038693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:57.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:57.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:57.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:57.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:57.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:57.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.039104 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:57.040704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.040704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:57.040704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:57.040704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:57.040704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:57.040704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:57.040704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:57.040704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:57.040704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:57.040704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:57.040704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:57.040704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:57.041312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:57.041312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:57.041312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:57.041312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:57.041312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:57.041312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:57.041312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:57.041312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:57.041312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:57.041312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:57.041312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:57.041312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:57.041312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:57.041596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.041596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:57.041596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.041596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:57.041596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:57.041596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:57.041596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:57.041596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:57.041596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:57.041596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:57.041596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:57.041596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:57.041596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:57.041596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:57.041596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:57.041869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:57.042169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:57.042169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:57.042169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:57.042169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:57.042169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.042169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:43:57.060583 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:43:57.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:43:57.067023 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:43:57.068003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:43:57.102148 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:43:57.102332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.105844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:43:57.105844 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:57.105844 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:43:57.105844 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:43:57.105844 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:57.105844 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:43:57.105844 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.105844 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:43:57.106437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.106437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:43:57.106437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:57.106437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:43:57.106437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:57.106437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:57.106437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:57.106437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:57.106437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:57.106437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:57.106437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:43:57.106437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:57.106785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:43:57.106785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:57.106785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:43:57.106785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:57.106785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:43:57.106785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:43:57.106785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:57.106785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:43:57.106785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:57.106785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:43:57.106785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:43:57.106785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:57.106785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:43:57.107083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.107083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:43:57.107154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.107154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:43:57.107154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:43:57.107154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:43:57.107154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:43:57.107154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:43:57.107154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:57.107154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:43:57.107154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:43:57.107154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:57.107154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:43:57.107154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:43:57.107154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:43:57.107154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:43:57.107154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:43:57.107154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:43:57.107522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:43:57.107975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:43:57.107975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:43:57.107975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:01.922370 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:01.924074 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:01.924074 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:01.924464 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:01.924681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:01.924805 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:01.924925 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:01.925038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:01.925152 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:01.925262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:01.925373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:01.926764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:01.926764 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:01.926764 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:01.926764 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:01.926764 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:01.926764 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:01.926764 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:01.926764 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:01.927846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:01.927846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:01.927846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:01.927846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:01.927846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:01.927846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:01.927846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:01.927846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:01.927846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:01.927846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:01.927846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:01.927846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:01.928281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:01.928281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:01.928281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:01.928281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:01.928281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:01.928281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:01.928281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:01.928281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:01.928281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:01.928281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:01.928281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:01.928281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:01.928281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:01.928608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:01.928608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:01.932329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:01.932329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:01.932329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:01.932329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:01.932329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:01.932329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:01.932329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:01.932329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:01.932329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:01.932329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:01.932329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:01.932329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:01.932329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:01.932329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:01.932329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:01.932329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:01.932746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:01.933164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:01.933164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:01.933164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:01.991236 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:01.991679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:01.991801 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:01.991904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:44:01.995028 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:01.995200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:01.995721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:01.995721 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:01.995721 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:01.995721 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:01.995721 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:01.995721 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:01.995721 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:01.995721 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:01.996374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:01.996374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:01.996374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:01.996374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:01.996374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:01.996374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:01.996374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:01.996374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:01.996374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:01.996374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:01.996374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:01.996374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:01.996374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:01.996374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:01.996374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:01.996834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:01.996834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:01.996834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:01.996834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:01.996834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:01.996834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:01.996834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:01.996834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:01.996834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:01.997240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:01.997240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:01.997240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:01.997240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:01.997240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:01.997240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:01.997240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:01.997240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:44:01.997240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:01.997240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:01.997240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:01.997240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:01.997588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:01.997588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:01.997588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:01.997588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:01.997588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:01.997588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:01.997588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:01.997588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:01.997588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:01.997588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:01.997588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:01.997588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:01.997588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:01.997588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:01.997923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:01.997923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:01.997923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:01.997923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:01.998026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:01.998026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:01.998026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:01.998026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:01.998026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:01.998026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:01.998026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:01.998026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:01.998026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:01.998026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:01.998026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:01.998026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:01.998026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:01.998026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:01.998026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:01.998026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:01.998397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:01.998849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:01.998849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:01.998849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:01.998849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:01.998849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:01.998849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:01.998849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:02.007652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:02.008193 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:02.008349 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:02.009146 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:02.009762 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:02.010028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.012646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:02.012646 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:02.012646 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:02.012646 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:02.012646 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:02.012646 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:02.012646 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.012646 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:02.014910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.014910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:02.014910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:02.014910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:02.014910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:02.014910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:02.014910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:02.014910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:02.014910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:02.014910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:02.014910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:02.014910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:02.014910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:02.014910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:02.014910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:02.015315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:02.015315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:02.015315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:02.015315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:02.015315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:02.015315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:02.015315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:02.015315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.015315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:02.018711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.007 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.018711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:02.018711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:02.018711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:02.018711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:02.018711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:02.018711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:02.018711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:02.018711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:02.018711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:02.018711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:02.019146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:02.019146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:02.019146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:02.019146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:02.019146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:02.019146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:02.019146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:02.019146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:02.019146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:02.019146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:02.019146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:02.019146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:02.019146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:02.019146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:02.019526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:02.019526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:02.019526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.019526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:02.020655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:02.021121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:02.021648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:02.021648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:02.021648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.021648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:02.035959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:02.037294 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:02.037459 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:02.037747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:02.038025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:02.038257 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:02.040727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.041304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:02.041304 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:02.041304 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:02.041304 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:02.041304 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:02.041304 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:02.041304 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.041304 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:02.042151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.042151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:02.042151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:02.042151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:02.042151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:02.042151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:02.042151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:02.042151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:02.042151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:02.042151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:02.042151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:02.042151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:02.042516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:02.042516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:02.042516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:02.042516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:02.042516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:02.042516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:02.042516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:02.042516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:02.042516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:02.042516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:02.042516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:02.042516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:02.042516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:02.042883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.042883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:02.043182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.043182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:02.043182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:02.043182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:02.043182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:02.043182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:02.043182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:02.043182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:02.043182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:02.043182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:02.043182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:02.043182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:02.043182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:02.043182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:02.043182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:02.043182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:02.043653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:02.044159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:02.044159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.044159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:02.061280 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:02.062652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:02.062796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:02.065576 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:02.128004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:02.128261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.129113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:02.129113 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:02.129113 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:02.129113 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:02.129113 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:02.129113 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:02.129113 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.129113 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:02.129829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.066 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.129829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:02.129829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:02.129829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:02.129829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:02.129829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:02.129829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:02.129829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:02.129829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:02.129829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:02.129829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:02.129829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:02.130122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:02.130122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:02.130122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:02.130122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:02.130122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:02.130122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:02.130122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:02.130122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:02.130122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:02.130122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:02.130122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:02.130122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:02.130122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:02.130384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.130384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:02.130679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.130679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:02.130679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:02.130679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:02.130679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:02.130679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:02.130679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:02.130679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:02.130679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:02.130679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:02.130679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:02.130679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:02.130679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:02.130679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:02.130679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:02.130679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:02.132038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:02.132509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:02.132509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:02.132509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:06.923223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:06.923691 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:06.923755 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:06.924617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:06.924874 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:06.924982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:06.925092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:06.925383 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:06.925383 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:06.925383 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:06.925526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:06.926095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:06.926095 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:06.926095 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:06.926095 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:06.926095 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:06.926095 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:06.926095 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:06.926095 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:06.927189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:06.927189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:06.927189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:06.927189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:06.927189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:06.927189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:06.927189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:06.927189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:06.927189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:06.927189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:06.927189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:06.927189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:06.927606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:06.927606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:06.927606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:06.927606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:06.927606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:06.927606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:06.927606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:06.927606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:06.927606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:06.927606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:06.927606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:06.927606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:06.927606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:06.927873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:06.927873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:06.927922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:06.927922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:06.927922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:06.927922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:06.927922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:06.927922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:06.927922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:06.927922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:06.927922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:06.927922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:06.927922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:06.927922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:06.927922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:06.927922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:06.927922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:06.927922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:06.928204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:06.928625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:06.928625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:06.928625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:06.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:06.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:06.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:06.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:44:06.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:06.992914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:06.993288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:06.993288 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:06.993288 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:06.993288 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:06.993288 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:06.993288 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:06.993288 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:06.993288 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:06.993959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:06.993959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:06.993959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:06.993959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:06.993959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:06.993959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:06.993959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:06.993959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:06.993959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:06.993959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:06.993959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:06.993959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:06.993959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:06.993959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:06.993959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:06.994490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:06.994490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:06.994490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:06.994490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:06.994490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:06.994490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:06.994490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:06.994490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:06.994490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:06.994885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:06.994885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:06.994885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:06.994885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:06.994885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:06.994885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:06.994885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:06.994885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:44:06.994885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:06.994885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:06.994885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:06.994885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:06.995244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:06.995244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:06.995244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:06.995244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:06.995244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:06.995244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:06.995244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:06.995244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:06.995244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:06.995244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:06.995244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:06.995244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:06.995244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:06.995244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:06.995651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:06.995651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:06.995651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:06.995651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:06.995651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:06.995651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:06.995651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:06.995651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:06.995651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:06.995651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:06.995651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:06.995651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:06.995651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:06.995651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:06.996043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:06.996573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:06.996573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:06.996573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:06.996573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:06.996573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:06.996573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:06.996573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:06.996573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:06.996573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:06.996573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:06.996573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:06.996573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:07.008160 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:07.008903 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:07.008903 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:07.008903 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:07.009183 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:07.009412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.009885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:07.009885 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:07.009885 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:07.009885 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:07.009885 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:07.009885 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:07.009885 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.009885 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:07.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:07.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:07.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:07.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:07.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:07.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:07.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:07.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:07.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:07.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:07.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:07.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:07.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:07.010427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:07.010876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:07.010876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:07.010876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:07.010876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:07.010876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:07.010876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:07.010876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:07.010876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.010876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:07.011107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.011107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:07.011107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:07.011107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:07.011107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:07.011107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:07.011107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:07.011107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:07.011107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:07.011107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:07.011107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:07.011417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:07.011417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:07.011417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:07.011417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:07.011417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:07.011417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:07.011417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:07.011417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:07.011417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:07.011417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:07.011417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:07.011417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:07.011417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:07.011417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:07.011820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:07.011820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:07.011820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.011820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:07.011820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.011820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:07.011820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:07.011820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:07.011820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:07.011820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:07.011820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:07.011820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:07.011820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:07.011820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:07.012194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:07.012735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:07.012735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:07.012735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:07.012735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:07.012735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:07.012735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:07.012735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:07.012735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:07.012735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:07.012735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.012735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:07.037136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:07.037519 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:07.037673 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:07.038820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:07.038820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:07.038820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:07.038820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.039105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:07.039105 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:07.039105 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:07.039105 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:07.039105 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:07.039105 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:07.039105 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.039105 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:07.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:07.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:07.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:07.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:07.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:07.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:07.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:07.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:07.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:07.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:07.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:07.040554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:07.040554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:07.040554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:07.040554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:07.040554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:07.040554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:07.040554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:07.040554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:07.040554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:07.040554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:07.040554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:07.040554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:07.040554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:07.041075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.041075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:07.041075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.041075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:07.041075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:07.041075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:07.041075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:07.041075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:07.041075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:07.041075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:07.041075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:07.041075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:07.041075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:07.041075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:07.041075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:07.042804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:07.043152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:07.043152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:07.043152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:07.043152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:07.043152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.043152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:07.062726 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:07.063000 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:07.063109 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:07.063830 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:07.093134 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:07.093134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.093307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:07.093307 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:07.093307 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:07.093307 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:07.093307 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:07.093307 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:07.093307 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.093307 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:07.094082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.094082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:07.094082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:07.094082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:07.094082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:07.094082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:07.094082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:07.094082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:07.094082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:07.094082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:07.094082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:07.094082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:07.094329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:07.094329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:07.094329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:07.094329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:07.094329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:07.094329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:07.094329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:07.094329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:07.094329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:07.094329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:07.094329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:07.094329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:07.094329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:07.094616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.094616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:07.094833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.094833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:07.094833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:07.094833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:07.094833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:07.094833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:07.094833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:07.094833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:07.094833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:07.094833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:07.094833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:07.094833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:07.094833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:07.094833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:07.094833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:07.094833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:07.095175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:07.095578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:07.095578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:07.095578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:11.923682 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:11.924195 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:11.924195 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:11.925142 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:11.925553 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:11.925802 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:11.925988 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:11.926837 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:11.926837 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:11.926837 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:11.926837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:11.927979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:11.927979 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:11.927979 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:11.927979 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:11.927979 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:11.927979 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:11.927979 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:11.927979 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:11.927979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:11.927979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:11.927979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:11.927979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:11.927979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:11.927979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:11.927979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:11.928382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:11.928382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:11.928382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:11.928382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:11.928382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:11.928382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:11.928382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:11.928382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:11.928382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:11.928382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:11.928382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:11.928382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:11.928382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:11.928382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:11.928774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:11.928774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:11.928774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:11.928774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:11.928774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:11.928774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:11.928774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:11.928774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:11.928774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:11.928774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:11.928774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:11.928774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:11.928774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:11.928774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:11.929144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:11.929788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:11.929788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:11.929788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:11.929788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:11.929788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:11.929788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:11.929788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:11.929788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:11.929788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:11.929788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:11.992590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:11.992590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:11.992590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:11.992813 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:44:11.993116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:11.993375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:11.993908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:11.993908 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:11.993908 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:11.993908 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:11.993908 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:11.993908 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:11.993908 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:11.993908 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:11.994532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:11.994532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:11.994532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:11.994532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:11.994532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:11.994532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:11.994532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:11.994532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:11.994532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:11.994532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:11.994532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:11.994532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:11.994532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:11.994532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:11.994532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:11.994864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:11.994864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:11.994864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:11.994864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:11.994864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:11.994864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:11.994864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:11.994864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:11.994864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:11.995332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:11.995332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:11.995332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:11.995332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:11.995332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:11.995332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:11.995332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:11.995332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:44:11.995332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:11.995332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:11.995332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:11.995332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:11.995586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:11.995586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:11.995586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:11.995586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:11.995586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:11.995586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:11.995586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:11.995586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:11.995586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:11.995586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:11.995586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:11.995586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:11.995586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:11.995586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:11.995803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:11.995803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:11.995803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:11.995803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:11.996261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:11.996261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:11.996261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:11.996261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:11.996261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:11.996261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:11.996261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:11.996261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:11.996261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:11.996261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:11.996261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:11.996261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:11.996261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:11.996261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:11.996261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:11.996261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:11.996594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:11.996901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:11.996901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:11.996901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:11.996901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:11.996901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:11.996901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:11.996901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:12.008603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:12.008869 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:12.009033 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:12.009147 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:12.009452 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:12.009703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.010088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:12.010088 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:12.010088 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:12.010088 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:12.010088 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:12.010088 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:12.010088 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.010088 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:12.010688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.010688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:12.010688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:12.010688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:12.010688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:12.010688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:12.010688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:12.010688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:12.010688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:12.010688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:12.010688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:12.010688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:12.010688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:12.010688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:12.010688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:12.011012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:12.011012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:12.011012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:12.011012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:12.011012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:12.011012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:12.011012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:12.011012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.011012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:12.011251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.011251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:12.011251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:12.011251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:12.011251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:12.011251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:12.011251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:12.011251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:12.011251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:12.011251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:12.011251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:12.011511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:12.011511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:12.011511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:12.011511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:12.011511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:12.011511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:12.011511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:12.011511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:12.011511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:12.011511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:12.011511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:12.011511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:12.011511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:12.011511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:12.011771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:12.011771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:12.011771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.011771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:12.011973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:12.012323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:12.012679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:12.012679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:12.012679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.012679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:12.037564 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:12.037987 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:12.038129 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:12.038329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:12.038859 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:12.039200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:12.039517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.040178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:12.040178 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:12.040178 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:12.040178 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:12.040178 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:12.040178 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:12.040178 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.040178 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:12.041030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.041030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:12.041030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:12.041030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:12.041030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:12.041030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:12.041030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:12.041030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:12.041030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:12.041030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:12.041030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:12.041030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:12.041357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:12.041357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:12.041357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:12.041357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:12.041357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:12.041357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:12.041357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:12.041357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:12.041357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:12.041357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:12.041357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:12.041357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:12.041357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:12.041773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.041773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:12.041888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.041888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:12.041888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:12.041888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:12.041888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:12.041888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:12.041888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:12.041888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:12.041888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:12.041888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:12.041888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:12.041888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:12.041888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:12.041888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:12.041888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:12.041888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:12.042256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:12.042876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:12.042876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.042876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:12.063226 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:12.063414 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:12.063511 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:12.064921 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:12.098761 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:12.098931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.099394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:12.099394 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:12.099394 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:12.099394 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:12.099394 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:12.099394 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:12.099394 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.099394 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:12.100229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.100229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:12.100229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:12.100229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:12.100229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:12.100229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:12.100229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:12.100229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:12.100229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:12.100229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:12.100229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:12.100229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:12.100562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:12.100562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:12.100562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:12.100562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:12.100562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:12.100562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:12.100562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:12.100562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:12.100562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:12.100562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:12.100562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:12.100562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:12.100562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:12.100839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.100839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:12.101204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.101204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:12.101204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:12.101204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:12.101204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:12.101204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:12.101204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:12.101204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:12.101204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:12.101204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:12.101204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:12.101204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:12.101204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:12.101204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:12.101204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:12.101204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:12.101643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:12.102021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:12.102021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:12.102021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:16.924187 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:16.924809 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:16.924809 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:16.926628 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:16.927151 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:16.927151 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:16.927151 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:16.927326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:16.927404 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:16.927586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:16.927698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:16.928166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:16.928166 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:16.928166 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:16.928166 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:16.928166 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:16.928166 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:16.928166 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:16.928166 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:16.929031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:16.929031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:16.929031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:16.929031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:16.929031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:16.929031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:16.929031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:16.929031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:16.929031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:16.929031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:16.929031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:16.929031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:16.929434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:16.929434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:16.929434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:16.929434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:16.929434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:16.929434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:16.929434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:16.929434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:16.929434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:16.929434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:16.929434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:16.929434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:16.929434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:16.929926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:16.929926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:16.930002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:16.930002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:16.930002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:16.930002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:16.930002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:16.930002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:16.930002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:16.930002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:16.930002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:16.930002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:16.930002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:16.930002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:16.930002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:16.930002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:16.930002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:16.930002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:16.930533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:16.931049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:16.931049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:16.931049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:16.993242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:16.994172 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:16.994293 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:16.994412 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:44:16.994750 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:16.995388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:16.995747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:16.995747 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:16.995747 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:16.995747 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:16.995747 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:16.995747 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:16.995747 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:16.995747 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:16.997139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:16.997139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:16.997139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:16.997139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:16.997139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:16.997139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:16.997139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:16.997139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:16.997139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:16.997139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:16.997139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:16.997139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:16.997139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:16.997139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:16.997139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:16.997526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:16.997526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:16.997526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:16.997526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:16.997526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:16.997526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:16.997526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:16.997526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:16.997526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:16.999552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:16.999552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:16.999552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:16.999552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:16.999552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:16.999552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:16.999552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:16.999552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:44:16.999552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:16.999552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:16.999552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:16.999552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:16.999847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:16.999847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:16.999847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:16.999847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:16.999847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:16.999847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:16.999847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:16.999847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:16.999847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:16.999847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:16.999847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:16.999847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:16.999847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:16.999847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:17.000159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:17.000159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:17.000159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.000159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:17.001919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.001919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:17.001919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:17.001919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:17.001919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:17.001919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:17.001919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.001919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:17.001919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:17.001919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.001919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:17.001919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:17.001919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.001919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:17.001919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:17.001919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:17.002251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:17.002740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:17.002740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:17.002740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:17.002740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:17.002740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:17.002740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.002740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:17.009110 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:17.009713 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:17.009852 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:17.009997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:17.011019 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:17.011019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.011248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:17.011248 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:17.011248 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:17.011248 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:17.011248 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:17.011248 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:17.011248 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.011248 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:17.012087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.012087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:17.012087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:17.012087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:17.012087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:17.012087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:17.012087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:17.012087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:17.012087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:17.012087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:17.012087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:17.012087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:17.012087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:17.012087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:17.012087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:17.012489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:17.012489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:17.012489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:17.012489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:17.012489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:17.012489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:17.012489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:17.012489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.012489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:17.013061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.013061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:17.013061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:17.013061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:17.013061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.013061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:17.013061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:17.013061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:17.013061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:17.013061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:17.013061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:17.013266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:17.013266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:17.013266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:17.013266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:17.013266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:17.013266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:17.013266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:17.013266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:17.013266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:17.013266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:17.013266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:17.013266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:17.013266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:17.013266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:17.013515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:17.013515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:17.013515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.013515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.014220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:17.014552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:17.014874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:17.014874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:17.014874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.014874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:17.038732 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:17.039145 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:17.039273 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:17.039474 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:17.040503 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:17.040503 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:17.040503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.040671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:17.040671 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:17.040671 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:17.040671 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:17.040671 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:17.040671 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:17.040671 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.040671 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:17.041410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.041410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:17.041410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:17.041410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:17.041410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.041410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:17.041410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:17.041410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:17.041410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:17.041410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:17.041410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:17.041410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:17.042100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:17.042100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:17.042100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:17.042100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:17.042100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:17.042100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:17.042100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:17.042100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:17.042100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:17.042100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:17.042100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:17.042100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:17.042100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:17.042512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.042512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:17.042512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.042512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:17.042512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:17.042512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:17.042512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:17.042512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:17.042512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.042512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:17.042512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:17.042512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.042512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:17.042512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:17.042512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:17.043042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:17.043353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:17.043353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:17.043353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:17.043353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:17.043353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.043353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:17.063559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:17.064000 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:17.064148 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:17.065021 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:17.111084 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:17.111084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.111249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:17.111249 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:17.111249 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:17.111249 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:17.111249 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:17.111249 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:17.111249 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.111249 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:17.112054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.112054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:17.112054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:17.112054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:17.112054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.112054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:17.112054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:17.112054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:17.112054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:17.112054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:17.112054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:17.112054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:17.112319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:17.112319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:17.112319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:17.112319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:17.112319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:17.112319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:17.112319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:17.112319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:17.112319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:17.112319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:17.112319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:17.112319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:17.112319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:17.112597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.112597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:17.113113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.113113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:17.113113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:17.113113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:17.113113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:17.113113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:17.113113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.113113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:17.113113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:17.113113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.113113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:17.113113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:17.113113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.113113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:17.113113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:17.113113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:17.113479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:17.113885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:17.113885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:17.113885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:21.924626 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:21.925603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:21.925706 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:21.926736 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:21.928427 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:21.928623 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:21.928748 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:21.928883 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:21.928984 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:21.929094 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:21.929200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:21.929696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:21.929696 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:21.929696 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:21.929696 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:21.929696 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:21.929696 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:21.929696 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:21.929696 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:21.930450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:21.930450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:21.930450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:21.930450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:21.930450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:21.930450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:21.930450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:21.930450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:21.930450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:21.930450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:21.930450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:21.930450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:21.930839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:21.930839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:21.930839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:21.930839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:21.930839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:21.930839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:21.930839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:21.930839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:21.930839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:21.930839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:21.930839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:21.930839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:21.930839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:21.931207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:21.931207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:21.931207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:21.931207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:21.931207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:21.931207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:21.931207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:21.931207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:21.931207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:21.931207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:21.931207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:21.931207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:21.931207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:21.931207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:21.931207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:21.931660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:21.932197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:21.932197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:21.932197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:21.932197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:21.932197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:21.932197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:21.994246 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:21.994479 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:21.994629 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:21.994766 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:44:21.995112 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:21.995300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:21.995756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:21.995756 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:21.995756 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:21.995756 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:21.995756 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:21.995756 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:21.995756 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:21.995756 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:21.996597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:21.996597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:21.996597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:21.996597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:21.996597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:21.996597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:21.996597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:21.996597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:21.996597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:21.996597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:21.996597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:21.996597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:21.996597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:21.996597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:21.996597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:21.996906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:21.996906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:21.996906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:21.996906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:21.996906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:21.996906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:21.996906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:21.996906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:21.996906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:21.997518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:21.997518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:21.997518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:21.997518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:21.997518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:21.997518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:21.997518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:21.997518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:44:21.997518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:21.997518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:21.997518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:21.997518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:21.997760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:21.997760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:21.997760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:21.997760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:21.997760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:21.997760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:21.997760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:21.997760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:21.997760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:21.997760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:21.997760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:21.997760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:21.997760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:21.997760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:21.998023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:21.998023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:21.998023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:21.998023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:21.998790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:21.998790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:21.998790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:21.998790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:21.998790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:21.998790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:21.998790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:21.998790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:21.998790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:21.998790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:21.998790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:21.998790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:21.998790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:21.998790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:21.998790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:21.998790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:21.999144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:21.999466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:21.999466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:21.999466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:21.999466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:21.999466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:21.999466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:21.999466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:22.009790 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:22.009978 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:22.010096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:22.012564 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:22.012564 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:22.012564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.012564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:22.012564 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:22.012564 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:22.012564 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:22.012564 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:22.012564 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:22.012564 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.012564 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:22.012564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:22.012929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:22.013332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:22.013332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:22.013332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:22.013332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:22.013332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.013332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:22.013332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.013332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:22.013332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:22.013332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:22.013332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:22.013332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:22.013332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:22.013704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:22.013704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:22.013704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:22.013704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:22.013704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:22.013704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:22.013704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:22.013704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:22.013704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:22.013704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:22.013704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:22.013704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:22.013704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:22.014022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:22.014022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:22.014022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:22.014022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:22.014022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:22.014022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:22.014022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:22.014022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.014022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:22.014022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.014022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:22.014022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:22.014327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.014821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:22.039156 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:22.039687 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:22.039812 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:22.039990 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:22.040289 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:22.040584 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:22.041120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.041656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:22.041656 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:22.041656 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:22.041656 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:22.041656 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:22.041656 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:22.041656 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.041656 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:22.042386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.042386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:22.042386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:22.042386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:22.042386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:22.042386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:22.042386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:22.042386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:22.042386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:22.042386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:22.042386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:22.042386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:22.042750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:22.042750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:22.042750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:22.042750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:22.042750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:22.042750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:22.042750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:22.042750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:22.042750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:22.042750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:22.042750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:22.042750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:22.042750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:22.043085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.043085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:22.043439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.043439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:22.043439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:22.043439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:22.043439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:22.043439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:22.043439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:22.043439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:22.043439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:22.043439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:22.043439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:22.043439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:22.043439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:22.043439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:22.043439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:22.043439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:22.043913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:22.044492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:22.044492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.044492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:22.064087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:22.064320 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:22.064506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:22.065187 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:22.076766 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=978,neutron:UPDATE=1,neutron:DELETE=13 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:44:22.110619 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:22.110753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.111226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:22.111226 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:22.111226 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:22.111226 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:22.111226 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:22.111226 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:22.111226 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.111226 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:22.111970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.111970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:22.111970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:22.111970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:22.111970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:22.111970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:22.111970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:22.111970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:22.111970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:22.111970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:22.111970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:22.111970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:22.112210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:22.112210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:22.112210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:22.112210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:22.112210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:22.112210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:22.112210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:22.112210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:22.112210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:22.112210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:22.112210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:22.112210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:22.112210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:22.112414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.112414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:22.112969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.112969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:22.112969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:22.112969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:22.112969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:22.112969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:22.112969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:22.112969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:22.112969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:22.112969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:22.112969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:22.112969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:22.112969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:22.112969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:22.112969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:22.112969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:22.113293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:22.113654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:22.113654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:22.113654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:26.925783 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:26.926899 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:26.926899 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:26.927273 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:26.927430 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:26.927597 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:26.927716 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:26.927821 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:26.927924 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:26.928073 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:26.928176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:26.928646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:26.928646 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:26.928646 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:26.928646 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:26.928646 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:26.928646 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:26.928646 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:26.928646 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:26.929274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:26.929274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:26.929274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:26.929274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:26.929274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:26.929274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:26.929274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:26.929274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:26.929274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:26.929274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:26.929274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:26.929274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:26.929770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:26.929770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:26.929770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:26.929770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:26.929770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:26.929770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:26.929770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:26.929770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:26.929770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:26.929770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:26.929770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:26.929770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:26.929770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:26.930127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:26.930127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:26.930420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:26.930420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:26.930420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:26.930420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:26.930420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:26.930420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:26.930420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:26.930420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:26.930420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:26.930420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:26.930420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:26.930420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:26.930420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:26.930420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:26.930420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:26.930420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:26.931519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:26.932601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:26.932601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:26.932601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:26.994742 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:26.995033 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:26.995137 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:26.995233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:44:26.995664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:26.995878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:26.997899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:26.997899 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:26.997899 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:26.997899 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:26.997899 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:26.997899 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:26.997899 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:26.997899 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:26.998496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:26.998496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:26.998496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:26.998496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:26.998496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:26.998496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:26.998496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:26.998496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:26.998496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:26.998496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:26.998496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:26.998496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:26.998496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:26.998496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:26.998496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:26.998786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:26.998786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:26.998786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:26.998786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:26.998786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:26.998786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:26.998786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:26.998786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:26.998786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:26.999044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:26.999044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:26.999044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:26.999044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:26.999044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:26.999044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:26.999044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:26.999044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:44:26.999044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:26.999044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:26.999044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:26.999044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:26.999614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:26.999614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:26.999614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:26.999614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:26.999614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:26.999614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:26.999614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:26.999614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:26.999614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:26.999614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:26.999614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:26.999614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:26.999614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:26.999614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:27.000240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:27.000240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:27.000240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.000240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:27.000240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.000240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:27.000240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:27.000240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:27.000240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:27.000240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:27.000240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.000240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:27.000240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:27.000240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:27.000517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:27.000856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:27.000856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:27.000856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:27.000856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:27.000856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:27.000856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:27.000856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:27.000856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:27.000856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:27.000856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:27.000856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.000856 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:27.010214 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:27.013237 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:27.013339 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:27.013419 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:27.013848 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:27.014085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.014572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:27.014572 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:27.014572 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:27.014572 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:27.014572 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:27.014572 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:27.014572 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.014572 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:27.015182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.015182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:27.015182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:27.015182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:27.015182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:27.015182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:27.015182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:27.015182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:27.015182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:27.015182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:27.015182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:27.015182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:27.015182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:27.015182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:27.015182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:27.015482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:27.015482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:27.015482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:27.015482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:27.015482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:27.015482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:27.015482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:27.015482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.015482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:27.015768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.015768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:27.015768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:27.015768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:27.015768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.015768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:27.015768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:27.015768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:27.015768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:27.015768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:27.015768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:27.016188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:27.016188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:27.016188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:27.016188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:27.016188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:27.016188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:27.016188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:27.016188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:27.016188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:27.016188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:27.016188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:27.016188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:27.016188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:27.016188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:27.016801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:27.016801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:27.016801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.016801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:27.016801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.016801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:27.016801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:27.016801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:27.016801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:27.016801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:27.016801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.016801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:27.016801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:27.016801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:27.017237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:27.017648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:27.017648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:27.017648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:27.017648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:27.017648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:27.017648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:27.017648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:27.017648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:27.017648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:27.017648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.017648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:27.039854 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:27.040120 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:27.040209 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:27.040362 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:27.040764 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:27.040976 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:27.041161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.041616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:27.041616 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:27.041616 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:27.041616 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:27.041616 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:27.041616 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:27.041616 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.041616 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:27.042267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.042267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:27.042267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:27.042267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:27.042267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.042267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:27.042267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:27.042267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:27.042267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:27.042267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:27.042267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:27.042267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:27.042667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:27.042667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:27.042667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:27.042667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:27.042667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:27.042667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:27.042667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:27.042667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:27.042667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:27.042667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:27.042667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:27.042667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:27.042667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:27.043011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.043011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:27.043011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.043011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:27.043011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:27.043011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:27.043011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:27.043011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:27.043011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.043011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:27.043011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:27.043011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.043011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:27.043011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:27.043011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:27.043394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:27.043881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:27.043881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:27.043881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:27.043881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:27.043881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.043881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:27.064441 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:27.064764 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:27.064914 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:27.065552 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:27.098824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:27.099397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.099998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:27.099998 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:27.099998 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:27.099998 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:27.099998 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:27.099998 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:27.099998 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.099998 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:27.100824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.100824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:27.100824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:27.100824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:27.100824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.100824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:27.100824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:27.100824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:27.100824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:27.100824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:27.100824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:27.100824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:27.101217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:27.101217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:27.101217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:27.101217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:27.101217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:27.101217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:27.101217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:27.101217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:27.101217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:27.101217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:27.101217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:27.101217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:27.101217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:27.101615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.101615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:27.101615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.101615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:27.101615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:27.101615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:27.101615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:27.101615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:27.101615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.101615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:27.101615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:27.101615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.101615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:27.101615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:27.101615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:27.102004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:27.102480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:27.102480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:27.102480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:27.102480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:27.102480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:27.102480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:31.926423 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:31.927085 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:31.927085 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:31.927610 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:31.927852 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:31.928024 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:31.928161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:31.928276 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:31.928401 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:31.928519 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:31.928657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:31.929176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:31.929176 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:31.929176 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:31.929176 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:31.929176 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:31.929176 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:31.929176 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:31.929176 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:31.929861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:31.929861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:31.929861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:31.929861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:31.929861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:31.929861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:31.929861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:31.929861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:31.929861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:31.929861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:31.929861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:31.929861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:31.930271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:31.930271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:31.930271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:31.930271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:31.930271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:31.930271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:31.930271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:31.930271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:31.930271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:31.930271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:31.930271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:31.930271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:31.930271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:31.930718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:31.930718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:31.930718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:31.930718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:31.930718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:31.930718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:31.930718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:31.930718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:31.930718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:31.930718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:31.930718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:31.930718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:31.930718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:31.930718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:31.930718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:31.931156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:31.931719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:31.931719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:31.931719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:31.931719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:31.931719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:31.931719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:31.995200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:31.995520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:31.995662 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:31.995776 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:44:31.996097 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:31.996258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:31.996680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:31.996680 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:31.996680 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:31.996680 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:31.996680 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:31.996680 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:31.996680 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:31.996680 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:31.997242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:31.997242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:31.997242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:31.997242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:31.997242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:31.997242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:31.997242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:31.997242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:31.997242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:31.997242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:31.997242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:31.997242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:31.997242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:31.997242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:31.997242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:31.997632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:31.997632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:31.997632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:31.997632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:31.997632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:31.997632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:31.997632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:31.997632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:31.997632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:31.997828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:31.997828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:31.997828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:31.997828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:31.997828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:31.997828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:31.997828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:31.997828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:44:31.997828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:31.997828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:31.997828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:31.997828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:31.998060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:31.998060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:31.998060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:31.998060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:31.998060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:31.998060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:31.998060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:31.998060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:31.998060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:31.998060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:31.998060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:31.998060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:31.998060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:31.998060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:31.998319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:31.998319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:31.998319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:31.998319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:31.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:31.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:31.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:31.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:31.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:31.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:31.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:31.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:31.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:31.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:31.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:31.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:31.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:31.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:31.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:31.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:31.999626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:32.000476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:32.000476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:32.000476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:32.000476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:32.000476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:32.000476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.000476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:32.013373 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:32.013722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:32.013860 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:32.014155 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:32.014488 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:32.014702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.015223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:32.015223 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:32.015223 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:32.015223 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:32.015223 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:32.015223 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:32.015223 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.015223 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:32.016101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.016101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:32.016101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:32.016101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:32.016101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:32.016101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:32.016101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:32.016101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:32.016101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:32.016101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:32.016101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:32.016101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:32.016101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:32.016101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:32.016101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:32.016560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:32.016560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:32.016560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:32.016560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:32.016560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:32.016560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:32.016560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:32.016560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.016560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:32.017178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.017178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:32.017178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:32.017178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:32.017178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:32.017178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:32.017178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:32.017178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:32.017178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:32.017178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:32.017178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:32.017562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:32.017562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:32.017562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:32.017562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:32.017562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:32.017562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:32.017562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:32.017562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:32.017562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:32.017562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:32.017562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:32.017562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:32.017562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:32.017562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:32.017967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:32.017967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:32.017967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.017967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:32.018356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:32.019243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:32.019804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:32.019804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:32.019804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.019804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:32.040274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:32.040631 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:32.040757 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:32.040953 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:32.041289 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:32.041509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:32.041716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.042125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:32.042125 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:32.042125 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:32.042125 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:32.042125 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:32.042125 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:32.042125 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.042125 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:32.042862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.042862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:32.042862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:32.042862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:32.042862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:32.042862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:32.042862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:32.042862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:32.042862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:32.042862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:32.042862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:32.042862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:32.043257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:32.043257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:32.043257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:32.043257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:32.043257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:32.043257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:32.043257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:32.043257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:32.043257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:32.043257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:32.043257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:32.043257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:32.043257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:32.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:32.043990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.043990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:32.043990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:32.043990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:32.043990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:32.043990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:32.043990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:32.043990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:32.043990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:32.043990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:32.043990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:32.043990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:32.043990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:32.043990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:32.043990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:32.043990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:32.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:32.045027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:32.045027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.045027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:32.064890 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:32.065130 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:32.065237 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:32.065869 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:32.102401 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:32.102645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.103175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:32.103175 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:32.103175 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:32.103175 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:32.103175 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:32.103175 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:32.103175 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.103175 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:32.103806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.103806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:32.103806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:32.103806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:32.103806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:32.103806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:32.103806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:32.103806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:32.103806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:32.103806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:32.103806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:32.103806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:32.104179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:32.104179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:32.104179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:32.104179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:32.104179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:32.104179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:32.104179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:32.104179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:32.104179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:32.104179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:32.104179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:32.104179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:32.104179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:32.104589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.104589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:32.104589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.104589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:32.104589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:32.104589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:32.104589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:32.104589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:32.104589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:32.104589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:32.104589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:32.104589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:32.104589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:32.104589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:32.104589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:32.105025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:32.105577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:32.105577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:32.105577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:32.105577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:32.105577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:32.105577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:36.926743 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:36.927297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:36.927297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:36.928197 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:36.928421 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:36.928606 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:36.928737 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:36.928855 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:36.928971 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:36.929080 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:36.929207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:36.929783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:36.929783 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:36.929783 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:36.929783 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:36.929783 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:36.929783 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:36.929783 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:36.929783 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:36.930613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:36.930613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:36.930613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:36.930613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:36.930613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:36.930613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:36.930613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:36.930613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:36.930613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:36.930613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:36.930613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:36.930613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:36.930878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:36.930878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:36.930878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:36.930878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:36.930878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:36.930878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:36.930878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:36.930878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:36.930878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:36.930878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:36.930878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:36.930878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:36.930878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:36.931092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:36.931092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:36.931471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:36.931471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:36.931471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:36.931471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:36.931471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:36.931471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:36.931471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:36.931471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:36.931471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:36.931471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:36.931471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:36.931471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:36.931471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:36.931471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:36.931471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:36.931471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:36.931804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:36.932139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:36.932139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:36.932139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:36.995722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:36.996089 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:36.996200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:36.996323 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:44:36.996920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:37.000557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:37.001057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.001057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:37.001057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:37.001057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:37.001057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:37.001057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:37.001057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:37.001057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:37.001057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:37.001057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:37.001057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:37.001057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:37.001057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:37.001057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:37.001057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.001057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:37.001407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.001407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:37.001407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:37.001407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:37.001407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.001407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:37.001407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:37.001407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:44:37.001407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:37.001407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:37.001407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.001407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:37.001674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:37.001674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:37.001674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.001674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:37.001674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:37.001674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:37.001674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:37.001674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:37.001674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:37.001674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:37.001674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:37.001674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:37.001674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:37.001674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:37.001938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:37.001938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:37.001938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.001938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:37.002510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.002510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:37.002510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:37.002510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:37.002510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:37.002510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:37.002510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.002510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:37.002510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:37.002510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.002510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:37.002510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:37.002510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.002510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:37.002510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:37.002510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:37.002822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:37.003180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:37.003180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:37.003180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:37.003180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:37.003180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:37.003180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.003180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:37.013807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:37.015739 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:37.015844 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:37.015946 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:37.016366 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:37.016732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.017230 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:37.017230 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:37.017230 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:37.017230 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:37.017230 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:37.017230 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:37.017230 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.017230 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:37.017847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.017847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:37.017847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:37.017847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:37.017847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.017847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:37.017847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:37.017847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:37.017847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.017847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:37.017847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:37.017847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:37.017847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:37.017847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:37.017847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:37.018334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:37.018334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:37.018334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:37.018334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:37.018334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:37.018334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:37.018334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:37.018334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.018334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:37.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:37.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:37.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:37.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:37.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:37.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:37.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:37.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:37.018991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:37.018991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:37.018991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.018991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:37.018991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:37.018991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:37.018991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:37.018991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:37.018991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:37.018991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:37.018991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:37.018991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:37.018991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:37.018991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:37.019294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:37.019294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:37.019294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.019294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.019380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:37.019723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:37.020031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:37.020031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:37.020031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.020031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:37.040809 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:37.041126 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:37.041253 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:37.041393 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:37.041941 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:37.042225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:37.042523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:37.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:37.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:37.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:37.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:37.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:37.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:37.043944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.043944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:37.043944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:37.043944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:37.043944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.043944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:37.043944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:37.043944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.043944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:37.043944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:37.043944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:37.043944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.044312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:37.044312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:37.044312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:37.044312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:37.044312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:37.044312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:37.044312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:37.044312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:37.044312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:37.044312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:37.044312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:37.044312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:37.044312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:37.044684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.044684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:37.044950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.044950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:37.044950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:37.044950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:37.044950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:37.044950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:37.044950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.044950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:37.044950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:37.044950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.044950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:37.044950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:37.044950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.044950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:37.044950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:37.044950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:37.045367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:37.045780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:37.045780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.045780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:37.065236 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:37.065644 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:37.065745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:37.066354 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:37.100466 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:37.100817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.101436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:37.101436 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:37.101436 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:37.101436 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:37.101436 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:37.101436 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:37.101436 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.101436 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:37.102346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.102346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:37.102346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:37.102346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:37.102346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.102346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:37.102346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:37.102346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.102346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:37.102346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:37.102346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:37.102346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.102749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:37.102749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:37.102749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:37.102749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:37.102749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:37.102749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:37.102749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:37.102749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:37.102749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:37.102749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:37.102749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:37.102749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:37.102749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:37.103085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.103085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:37.103085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.103085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:37.103085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:37.103085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:37.103085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:37.103085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:37.103085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.103085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:37.103085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:37.103085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.103085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:37.103085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:37.103085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:37.103405 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:37.103730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:37.103730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:37.103730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:37.103730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:37.103730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:37.103730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:41.928502 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:41.929298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:41.929298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:41.930188 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:41.930478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:41.930610 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:41.930743 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:41.930869 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:41.931010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-a6b960b9-a555-46bb-bb6b-3e12a9085233, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:41.931125 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:41.931246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:41.931906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:41.931906 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:41.931906 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:41.931906 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:41.931906 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:41.931906 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:41.931906 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:41.931906 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:41.932677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:41.932677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:41.932677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:41.932677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:41.932677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:41.932677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:41.932677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:41.932677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:41.932677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:41.932677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:41.932677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:41.932677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:41.932970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:41.932970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:41.932970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:41.932970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:41.932970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:41.932970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:41.932970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:41.932970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:41.932970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:41.932970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:41.932970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:41.932970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:41.932970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:41.933296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:41.933296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:41.933689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:41.933689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:41.933689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:41.933689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:41.933689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:41.933689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:41.933689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:41.933689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:41.933689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:41.933689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:41.933689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:41.933689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:41.933689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:41.933689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:41.933689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:41.933689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:41.934489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:41.935347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:41.935347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:41.935347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:41.996230 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:41.996670 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:41.996890 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:41.997041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:44:41.997562 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:41.997750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:41.998290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:41.998290 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:41.998290 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:41.998290 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:41.998290 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:41.998290 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:41.998290 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:41.998290 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:41.998996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:41.998996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:41.998996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:41.998996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:41.998996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:41.998996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:41.998996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:41.998996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:41.998996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:41.998996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:41.998996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:41.998996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:41.998996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:41.998996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:41.998996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:41.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:41.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:41.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:41.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:41.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:41.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:41.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:41.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:41.999332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:41.999800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:41.999800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:41.999800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:41.999800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:41.999800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:41.999800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:41.999800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:41.999800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:44:41.999800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:41.999800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:41.999800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:41.999800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:42.000374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:42.000374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:42.000374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.000374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:42.000374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:42.000374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:42.000374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:42.000374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:42.000374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:42.000374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:42.000374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:42.000374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:42.000374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:42.000374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:42.000911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:42.000911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:42.000911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.000911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:42.000911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.000911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:42.000911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:42.000911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:42.000911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:42.000911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:42.000911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.000911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:42.000911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:42.000911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:42.001280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:42.002122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:42.002122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:42.002122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:42.002122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:42.002122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:42.002122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:42.002122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:42.002122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:42.002122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:42.002122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:42.002122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.002122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:42.014340 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:42.017365 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:42.017561 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:42.017719 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:42.018740 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:42.019127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.019789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:42.019789 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:42.019789 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:42.019789 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:42.019789 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:42.019789 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:42.019789 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.019789 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:42.020723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.020723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:42.020723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:42.020723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:42.020723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.020723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:42.020723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:42.020723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:42.020723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.020723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:42.020723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:42.020723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:42.020723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:42.020723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:42.020723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:42.021102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:42.021102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:42.021102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:42.021102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:42.021102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:42.021102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:42.021102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:42.021102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.021102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:42.024570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.024570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:42.024570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:42.024570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:42.024570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.024570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:42.024570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:42.024570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:42.024570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:42.024570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.024570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:42.024979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:42.024979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:42.024979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.024979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:42.024979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:42.024979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:42.024979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:42.024979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:42.024979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:42.024979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:42.024979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:42.024979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:42.024979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:42.024979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:42.025327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:42.025327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:42.025327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.025327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.025457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:42.025932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:42.026409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:42.026409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:42.026409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.026409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:42.041257 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:42.041581 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:42.041714 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:42.041930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:42.042326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:42.042533 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:42.042750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.043223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:42.043223 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:42.043223 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:42.043223 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:42.043223 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:42.043223 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:42.043223 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.043223 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:42.043924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.043924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:42.043924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:42.043924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:42.043924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.043924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:42.043924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:42.043924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.043924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:42.043924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:42.043924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:42.043924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.044307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:42.044307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:42.044307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:42.044307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:42.044307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:42.044307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:42.044307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:42.044307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:42.044307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:42.044307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:42.044307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:42.044307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:42.044307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:42.044676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.044676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:42.045108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.045108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:42.045108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:42.045108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:42.045108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:42.045108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:42.045108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.045108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:42.045108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:42.045108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.045108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:42.045108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:42.045108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.045108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:42.045108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:42.045108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:42.046105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:42.047025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:42.047025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.047025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:42.065704 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:42.066005 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:42.066094 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:42.066679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:42.097422 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:42.099558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.099558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:42.099558 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:42.099558 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:42.099558 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:42.099558 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:42.099558 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:42.099558 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.099558 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:42.099558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.099558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:42.099558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:42.099558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:42.099558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.099558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:42.100000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:42.100000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.100000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:42.100000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:42.100000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:42.100000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.100000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:42.100000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:42.100000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:42.100000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:42.100000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:42.100000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:42.100000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:42.100000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:42.100313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:42.100313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:42.100313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:42.100313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:42.100313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:42.100313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.100313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:42.100313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.100313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:42.100313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:42.100313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:42.100313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:42.100313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:42.100659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:42.101096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:42.101096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:42.101096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:42.101096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:42.101096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:42.101096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:42.101096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:42.101096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:42.101096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:42.101096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:42.101096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:42.101096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:46.928996 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:46.929597 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:46.929597 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:46.930180 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:46.930343 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:46.930552 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:46.930678 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:46.930831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:46.931002 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:46.931137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:46.932114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:46.932114 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:46.932114 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:46.932114 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:46.932114 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:46.932114 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:46.932114 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:46.932114 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:46.932498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:46.932498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:46.932498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:46.932498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:46.932498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:46.932498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:46.932498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:46.932498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:46.932498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:46.932498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:46.932498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:46.932498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:46.932981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:46.932981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:46.932981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:46.932981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:46.932981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:46.932981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:46.932981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:46.932981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:46.932981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:46.932981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:46.932981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:46.932981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:46.932981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:46.933281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:46.933281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:46.933281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:46.933281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:46.933281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:46.933281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:46.933281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:46.933281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:46.933281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:46.933281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:46.933281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:46.933281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:46.933281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:46.933281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:46.933281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:46.933632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:46.934069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:46.934069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:46.934069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:46.934069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:46.934069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:46.934069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:46.997217 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:46.997934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:46.998049 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:46.998156 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:44:46.998731 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:46.999023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:46.999662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:46.999662 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:46.999662 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:46.999662 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:46.999662 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:46.999662 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:46.999662 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:46.999662 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:47.000820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.000820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:47.000820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:47.000820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:47.000820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.000820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:47.000820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:47.000820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:47.000820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.000820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:47.000820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:47.000820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:47.000820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:47.000820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:47.000820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:47.001278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:47.001278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:47.001278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:47.001278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:47.001278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:47.001278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:47.001278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:47.001278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.001278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:47.001908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.001908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:47.001908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:47.001908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:47.001908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.001908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:47.001908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:47.001908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:44:47.001908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:47.001908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:47.001908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.001908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:47.002288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:47.002288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:47.002288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.002288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:47.002288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:47.002288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:47.002288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:47.002288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:47.002288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:47.002288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:47.002288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:47.002288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:47.002288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:47.002288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:47.002744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:47.002744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:47.002744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.002744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:47.002744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.002744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:47.002744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:47.002744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:47.002744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:47.002744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:47.002744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.002744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:47.002744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:47.002744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:47.003167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:47.003741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:47.003741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:47.003741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:47.003741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:47.003741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:47.003741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:47.003741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:47.003741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:47.003741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:47.003741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:47.003741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.003741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:47.015925 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:47.016165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:47.016240 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:47.016351 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:47.016895 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:47.017510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.017872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:47.017872 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:47.017872 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:47.017872 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:47.017872 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:47.017872 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:47.017872 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.017872 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:47.018458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.018458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:47.018458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:47.018458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:47.018458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.018458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:47.018458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:47.018458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:47.018458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.018458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:47.018458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:47.018458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:47.018458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:47.018458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:47.018458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:47.018931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:47.018931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:47.018931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:47.018931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:47.018931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:47.018931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:47.018931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:47.018931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.018931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:47.019206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.019206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:47.019206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:47.019206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:47.019206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.019206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:47.019206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:47.019206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:47.019206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:47.019206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.019206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:47.019562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:47.019562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:47.019562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.019562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:47.019562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:47.019562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:47.019562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:47.019562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:47.019562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:47.019562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:47.019562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:47.019562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:47.019562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:47.019562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:47.019970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:47.019970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:47.019970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.019970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:47.019970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.019970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:47.019970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:47.019970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:47.019970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:47.019970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:47.019970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.019970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:47.019970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:47.019970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:47.020370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:47.020946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:47.020946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:47.020946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:47.020946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:47.020946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:47.020946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:47.020946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:47.020946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:47.020946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:47.020946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.020946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:47.041791 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:47.041932 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:47.042020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:47.042165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:47.042717 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:47.043000 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:47.043239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.043845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:47.043845 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:47.043845 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:47.043845 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:47.043845 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:47.043845 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:47.043845 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.043845 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:47.044556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.044556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:47.044556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:47.044556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:47.044556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.044556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:47.044556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:47.044556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.044556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:47.044556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:47.044556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:47.044556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.044956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:47.044956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:47.044956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:47.044956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:47.044956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:47.044956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:47.044956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:47.044956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:47.044956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:47.044956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:47.044956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:47.044956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:47.044956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:47.045350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.045350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:47.045425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.045425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:47.045425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:47.045425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:47.045425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:47.045425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:47.045425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.045425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:47.045425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:47.045425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.045425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:47.045425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:47.045425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.045425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:47.045425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:47.045425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:47.046718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:47.046718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.046718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:47.066127 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:47.066480 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:47.066616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:47.067266 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:47.096400 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:47.096617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.097138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:47.097138 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:47.097138 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:47.097138 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:47.097138 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:47.097138 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:47.097138 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.097138 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:47.097768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.097768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:47.097768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:47.097768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:47.097768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.097768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:47.097768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:47.097768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.097768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:47.097768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:47.097768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:47.097768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.098190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:47.098190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:47.098190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:47.098190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:47.098190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:47.098190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:47.098190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:47.098190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:47.098190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:47.098190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:47.098190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:47.098190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:47.098190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:47.098620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.098620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:47.098620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.098620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:47.098620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:47.098620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:47.098620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:47.098620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:47.098620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.098620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:47.098620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:47.098620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.098620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:47.098620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:47.098620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:47.099048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:47.099566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:47.099566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:47.099566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:47.099566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:47.099566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:47.099566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:51.929559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:51.930108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:51.930260 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:51.931215 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:51.931414 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:51.931622 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:51.931743 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:51.931853 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:51.931953 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:51.932056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:51.933912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:51.933912 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:51.933912 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:51.933912 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:51.933912 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:51.933912 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:51.933912 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:51.933912 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:51.935008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:51.935008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:51.935008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:51.935008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:51.935008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:51.935008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:51.935008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:51.935008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:51.935008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:51.935008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:51.935008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:51.935008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:51.935302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:51.935302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:51.935302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:51.935302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:51.935302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:51.935302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:51.935302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:51.935302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:51.935302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:51.935302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:51.935302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:51.935302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:51.935302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:51.935793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:51.935793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:51.935870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:51.935870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:51.935870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:51.935870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:51.935870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:51.935870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:51.935870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:51.935870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:51.935870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:51.935870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:51.935870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:51.935870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:51.935870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:51.935870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:51.935870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:51.935870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:51.936176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:51.936866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:51.936866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:51.936866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:51.997643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:51.998019 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:51.998166 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:51.998311 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:44:51.998781 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:51.998986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:51.999558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:51.999558 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:51.999558 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:51.999558 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:51.999558 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:51.999558 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:51.999558 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:51.999558 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:52.000312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.000312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:52.000312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:52.000312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:52.000312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.000312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:52.000312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:52.000312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:52.000312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.000312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:52.000312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:52.000312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:52.000312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:52.000312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:52.000312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.000779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:52.001229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.001229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:52.001229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:52.001229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:52.001229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.001229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:52.001229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:52.001229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:44:52.001229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:52.001229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:52.001229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.001229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:52.001605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:52.001605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:52.001605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.001605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:52.001605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:52.001605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:52.001605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:52.001605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:52.001605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:52.001605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:52.001605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:52.001605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:52.001605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:52.001605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:52.001944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:52.001944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:52.001944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.001944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:52.002311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.002311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:52.002311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:52.002311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:52.002311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:52.002311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:52.002311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.002311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:52.002311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:52.002311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.002311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:52.002311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:52.002311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.002311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:52.002311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:52.002311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:52.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:52.003159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:52.003159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:52.003159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:52.003159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:52.003159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:52.003159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.003159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:52.016281 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:52.016698 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:52.016864 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:52.017029 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:52.017379 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:52.017652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.018106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:52.018106 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:52.018106 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:52.018106 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:52.018106 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:52.018106 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:52.018106 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.018106 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:52.018828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.018828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:52.018828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:52.018828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:52.018828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.018828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:52.018828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:52.018828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:52.018828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.018828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:52.018828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:52.018828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:52.018828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:52.018828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:52.018828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:52.019217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:52.019217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:52.019217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:52.019217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:52.019217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:52.019217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:52.019217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:52.019217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.019217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:52.019830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.019830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:52.019830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:52.019830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:52.019830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.019830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:52.019830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:52.019830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:52.019830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:52.019830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.019830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:52.020165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:52.020165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:52.020165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.020165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:52.020165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:52.020165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:52.020165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:52.020165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:52.020165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:52.020165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:52.020165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:52.020165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:52.020165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:52.020165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:52.020898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:52.020898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:52.020898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.020898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.021211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:52.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:52.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:52.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:52.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:52.042045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:52.042405 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:52.042569 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:52.042754 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:52.043044 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:52.043279 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:52.043527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:52.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:52.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:52.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:52.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:52.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:52.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:52.044796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.044796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:52.044796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:52.044796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:52.044796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.044796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:52.044796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:52.044796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.044796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:52.044796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:52.044796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:52.044796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.045147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:52.045147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:52.045147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:52.045147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:52.045147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:52.045147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:52.045147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:52.045147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:52.045147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:52.045147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:52.045147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:52.045147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:52.045147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:52.045528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.045528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:52.045963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.045963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:52.045963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:52.045963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:52.045963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:52.045963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:52.045963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.045963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:52.045963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:52.045963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.045963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:52.045963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:52.045963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.045963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:52.045963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:52.045963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:52.046312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:52.046633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:52.046633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.046633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:52.066601 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:52.066975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:52.067121 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:52.067865 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:52.097893 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:52.098039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.098482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:52.098482 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:52.098482 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:52.098482 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:52.098482 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:52.098482 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:52.098482 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.098482 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:52.099378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.099378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:52.099378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:52.099378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:52.099378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.099378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:52.099378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:52.099378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.099378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:52.099378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:52.099378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:52.099378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.100011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:52.100011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:52.100011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:52.100011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:52.100011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:52.100011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:52.100011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:52.100011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:52.100011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:52.100011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:52.100011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:52.100011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:52.100011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:52.100714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.100714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:52.100714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.100714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:52.100714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:52.100714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:52.100714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:52.100714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:52.100714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.100714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:52.100714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:52.100714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.100714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:52.100714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:52.100714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:52.101198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:52.101690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:52.101690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:52.101690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:52.101690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:52.101690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:52.101690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:56.930015 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:56.930441 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:56.930441 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:56.931497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:56.931639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:56.931750 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:56.931877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:56.931992 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:56.932108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:56.932188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:56.932798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:56.932798 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:56.932798 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:56.932798 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:56.932798 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:56.932798 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:56.932798 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:56.932798 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:56.933437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:56.933437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:56.933437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:56.933437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:56.933437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:56.933437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:56.933437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:56.933437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:56.933437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:56.933437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:56.933437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:56.933437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:56.933741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:56.933741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:56.933741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:56.933741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:56.933741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:56.933741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:56.933741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:56.933741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:56.933741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:56.933741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:56.933741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:56.933741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:56.933741 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:56.933964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:56.933964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:56.934192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:56.934192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:56.934192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:56.934192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:56.934192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:56.934192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:56.934192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:56.934192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:56.934192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:56.934192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:56.934192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:56.934192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:56.934192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:56.934192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:56.934192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:56.934192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:56.934517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:56.934866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:56.934866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:56.934866 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:56.998089 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:56.998553 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:56.998683 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:56.998826 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:44:56.999148 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:56.999342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:56.999867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:56.999867 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:56.999867 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:56.999867 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:56.999867 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:56.999867 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:56.999867 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:56.999867 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:57.000552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.000552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:57.000552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:57.000552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:57.000552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.000552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:57.000552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:57.000552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:57.000552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.000552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:57.000552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:57.000552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:57.000552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:57.000552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:57.000552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:57.000965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:57.000965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:57.000965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:57.000965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:57.000965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:57.000965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:57.000965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:57.000965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.000965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:57.001678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.001678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:57.001678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:57.001678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:57.001678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.001678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:57.001678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:57.001678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:44:57.001678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:57.001678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:57.001678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.001678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:57.002495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:57.002495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:57.002495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.002495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:57.002495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:57.002495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:57.002495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:57.002495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:57.002495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:57.002495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:57.002495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:57.002495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:57.002495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:57.002495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:57.002953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:57.002953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:57.002953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.002953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:57.002953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.002953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:57.002953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:57.002953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:57.002953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:57.002953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:57.002953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.002953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:57.002953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:57.002953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:57.003203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:57.003669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:57.003669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:57.003669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:57.003669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:57.003669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:57.003669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:57.003669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:57.003669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:57.003669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:57.003669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:57.003669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.003669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:57.016759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:57.017089 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:57.017245 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:57.017391 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:57.017707 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:57.017939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.018416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:57.018416 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:57.018416 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:57.018416 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:57.018416 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:57.018416 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:57.018416 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.018416 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:57.019127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.019127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:44:57.019127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:57.019127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:57.019127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.019127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:57.019127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:57.019127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:44:57.019127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.019127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:44:57.019127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:57.019127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:44:57.019127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:57.019127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:44:57.019127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:44:57.019522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:57.019522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:44:57.019522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:57.019522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:44:57.019522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:44:57.019522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:57.019522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:44:57.019522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.019522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:44:57.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:57.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:57.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:57.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:57.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:57.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:57.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:57.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.020104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:57.020756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:57.020756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:57.020756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.020756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:57.020756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:57.020756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:57.020756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:57.020756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:57.020756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:57.020756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:57.020756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:57.020756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:57.020756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:57.020756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:57.021560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:57.021560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:57.021560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.021560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:57.021560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.021560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:57.021560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:57.021560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:57.021560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:57.021560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:57.021560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.021560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:57.021560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:57.021560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:57.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:57.022132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:57.022132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:57.022132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:57.022132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:57.022132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:57.022132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:57.022132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:57.022132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:57.022132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:57.022132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.022132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:57.042507 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:57.042836 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:57.043106 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:57.043851 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:57.044242 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:57.044456 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:57.044676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.045144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:57.045144 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:57.045144 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:57.045144 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:57.045144 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:57.045144 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:57.045144 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.045144 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:57.045949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.045949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:57.045949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:57.045949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:57.045949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.045949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:57.045949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:57.045949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.045949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:57.045949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:57.045949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:57.045949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.046659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:57.046659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:57.046659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:57.046659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:57.046659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:57.046659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:57.046659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:57.046659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:57.046659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:57.046659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:57.046659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:57.046659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:57.046659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:57.047414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.047414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:57.047414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.047414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:57.047414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:57.047414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:57.047414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:57.047414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:57.047414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.047414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:57.047414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:57.047414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.047414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:57.047414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:57.047414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:57.047783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:57.048248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:57.048248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:57.048248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:57.048248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:57.048248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.048248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:44:57.067031 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:44:57.067315 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:44:57.067442 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:44:57.068313 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:44:57.103119 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:44:57.103338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.103833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:44:57.103833 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:57.103833 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:44:57.103833 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:44:57.103833 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:57.103833 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:44:57.103833 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.103833 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:44:57.104902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.104902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:44:57.104902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:57.104902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:44:57.104902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.104902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:57.104902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:57.104902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.104902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:57.104902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:57.104902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:44:57.104902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.105165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:44:57.105165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:57.105165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:44:57.105165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:57.105165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:44:57.105165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:44:57.105165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:57.105165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:44:57.105165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:57.105165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:44:57.105165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:44:57.105165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:57.105165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:44:57.105380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.105380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:44:57.106685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.106685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:44:57.106685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:44:57.106685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:44:57.106685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:44:57.106685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:44:57.106685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.106685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:44:57.106685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:44:57.106685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.106685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:44:57.106685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:44:57.106685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.106685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:44:57.106685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:44:57.106685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:44:57.107196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:44:57.107729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:44:57.107729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:44:57.107729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:01.930650 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:01.932085 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:01.932085 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:01.932670 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:01.932859 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:01.933556 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:01.933556 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:01.933556 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:01.933556 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:01.933706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:01.934572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:01.934572 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:01.934572 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:01.934572 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:01.934572 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:01.934572 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:01.934572 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:01.934572 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:01.934932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:01.934932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:01.934932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:01.934932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:01.934932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:01.934932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:01.934932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:01.934932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:01.934932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:01.934932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:01.934932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:01.934932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:01.935387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:01.935387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:01.935387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:01.935387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:01.935387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:01.935387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:01.935387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:01.935387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:01.935387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:01.935387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:01.935387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:01.935387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:01.935387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:01.935699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:01.935699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:01.936714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:01.936714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:01.936714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:01.936714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:01.936714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:01.936714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:01.936714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:01.936714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:01.936714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:01.936714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:01.936714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:01.936714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:01.936714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:01.936714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:01.936714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:01.936714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:01.937163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:01.937648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:01.937648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:01.937648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:01.998669 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:01.998868 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:01.998980 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:01.999315 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:45:01.999674 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:01.999875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.000460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:02.000460 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:02.000460 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:02.000460 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:02.000460 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:02.000460 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:02.000460 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.000460 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:02.001266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.001266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:02.001266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:02.001266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:02.001266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.001266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:02.001266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:02.001266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:02.001266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.001266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:02.001266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:02.001266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:02.001266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:02.001266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:02.001266 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:02.001804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:02.001804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:02.001804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:02.001804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:02.001804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:02.001804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:02.001804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:02.001804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.001804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:02.002302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.002302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:02.002302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:02.002302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:02.002302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.002302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:02.002302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:02.002302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:45:02.002302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:02.002302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:02.002302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.002302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:02.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:02.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:02.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:02.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:02.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:02.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:02.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:02.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:02.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:02.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:02.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:02.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:02.002715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:02.003140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:02.003140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:02.003140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.003140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:02.003374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.003374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:02.003374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:02.003374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:02.003374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:02.003374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:02.003374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.003374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:02.003374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:02.003374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.003374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:02.003374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:02.003374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.003374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:02.003374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:02.003374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:02.003785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:02.004147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:02.004147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:02.004147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:02.004147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:02.004147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:02.004147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.004147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:02.017249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:02.017555 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:02.017692 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:02.017834 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:02.018374 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:02.018635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.019031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:02.019031 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:02.019031 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:02.019031 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:02.019031 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:02.019031 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:02.019031 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.019031 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:02.019749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.019749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:02.019749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:02.019749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:02.019749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.019749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:02.019749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:02.019749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:02.019749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.019749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:02.019749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:02.019749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:02.019749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:02.019749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:02.019749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:02.020153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:02.020153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:02.020153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:02.020153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:02.020153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:02.020153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:02.020153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:02.020153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.020153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:02.020728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.020728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:02.020728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:02.020728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:02.020728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.020728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:02.020728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:02.020728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:02.020728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:02.020728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.020728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:02.021177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:02.021177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:02.021177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.021177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:02.021177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:02.021177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:02.021177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:02.021177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:02.021177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:02.021177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:02.021177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:02.021177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:02.021177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:02.021177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:02.021673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:02.021673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:02.021673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.021673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.021910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:02.022281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:02.022675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:02.022675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:02.022675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.022675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:02.042945 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:02.043239 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:02.046299 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:02.046299 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:02.046299 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:02.046299 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:02.046299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.046758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:02.046758 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:02.046758 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:02.046758 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:02.046758 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:02.046758 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:02.046758 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.046758 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:02.047513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.047513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:02.047513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:02.047513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:02.047513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.047513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:02.047513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:02.047513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.047513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:02.047513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:02.047513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:02.047513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.047935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:02.047935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:02.047935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:02.047935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:02.047935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:02.047935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:02.047935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:02.047935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:02.047935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:02.047935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:02.047935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:02.047935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:02.047935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:02.048326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.048326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:02.048637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.048637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:02.048637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:02.048637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:02.048637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:02.048637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:02.048637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.048637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:02.048637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:02.048637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.048637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:02.048637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:02.048637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.048637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:02.048637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:02.048637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:02.049121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:02.049726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:02.049726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.049726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:02.067402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:02.067749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:02.067863 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:02.068421 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:02.107113 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:02.107113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.107113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:02.107113 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:02.107113 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:02.107113 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:02.107113 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:02.107113 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:02.107113 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.107113 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:02.107327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.039 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.107327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:02.107327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:02.107327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:02.107327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.107327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:02.107327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:02.107327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.107327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:02.107327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:02.107327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:02.107327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.107645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:02.107645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:02.107645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:02.107645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:02.107645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:02.107645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:02.107645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:02.107645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:02.107645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:02.107645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:02.107645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:02.107645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:02.107645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:02.107910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.107910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:02.107910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.107910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:02.107910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:02.107910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:02.107910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:02.107910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:02.107910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.107910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:02.107910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:02.107910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.107910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:02.107910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:02.107910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:02.108216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:02.108576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:02.108576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:02.108576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:02.108576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:02.108576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:02.108576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:06.931557 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:06.931930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:06.932011 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:06.932904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:06.933098 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:06.933274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:06.933436 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:06.933627 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:06.933795 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:06.933927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:06.934515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:06.934515 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:06.934515 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:06.934515 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:06.934515 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:06.934515 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:06.934515 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:06.934515 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:06.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:06.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:06.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:06.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:06.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:06.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:06.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:06.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:06.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:06.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:06.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:06.935277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:06.935497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:06.935497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:06.935497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:06.935497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:06.935497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:06.935497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:06.935497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:06.935497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:06.935497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:06.935497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:06.935497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:06.935497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:06.935497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:06.935759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:06.935759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:06.936336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:06.936336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:06.936336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:06.936336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:06.936336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:06.936336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:06.936336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:06.936336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:06.936336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:06.936336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:06.936336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:06.936336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:06.936336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:06.936336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:06.936336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:06.936336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:06.936625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:06.937102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:06.937102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:06.937102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:06.998979 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:06.999281 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:06.999398 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:06.999530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:45:06.999896 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:07.000064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.000495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:07.000495 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:07.000495 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:07.000495 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:07.000495 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:07.000495 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:07.000495 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.000495 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:07.001202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.001202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:07.001202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:07.001202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:07.001202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.001202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:07.001202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:07.001202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:07.001202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.001202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:07.001202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:07.001202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:07.001202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:07.001202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:07.001202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:07.001477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:07.001477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:07.001477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:07.001477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:07.001477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:07.001477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:07.001477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:07.001477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.001477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:07.001979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.001979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:07.001979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:07.001979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:07.001979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.001979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:07.001979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:07.001979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:45:07.001979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:07.001979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:07.001979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.001979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:07.002188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:07.002188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:07.002188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.002188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:07.002188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:07.002188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:07.002188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:07.002188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:07.002188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:07.002188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:07.002188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:07.002188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:07.002188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:07.002188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:07.002414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:07.002414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:07.002414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.002414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:07.002810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.002810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:07.002810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:07.002810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:07.002810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:07.002810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:07.002810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.002810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:07.002810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:07.002810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.002810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:07.002810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:07.002810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.002810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:07.002810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:07.002810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:07.003087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:07.003368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:07.003368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:07.003368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:07.003368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:07.003368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:07.003368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.003368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:07.017791 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:07.021563 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:07.021563 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:07.021563 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:07.021563 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:07.021563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.021563 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:07.021563 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:07.021563 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:07.021563 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:07.021875 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:07.021875 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:07.021875 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.021875 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:07.021875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.021875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:07.021875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:07.021875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:07.021875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.021875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:07.021875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:07.021875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:07.021875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.021875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:07.021875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:07.021875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:07.022148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:07.022148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:07.022148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:07.022148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:07.022148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:07.022148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:07.022148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:07.022148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:07.022148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:07.022148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:07.022148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.022148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:07.022148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.022148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:07.022366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:07.022366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:07.022366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.022366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:07.022366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:07.022366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:07.022366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:07.022366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.022366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:07.022366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:07.022366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:07.022366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.022366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:07.022957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:07.022957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:07.022957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:07.022957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:07.022957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:07.022957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:07.022957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:07.022957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:07.022957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:07.022957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:07.022957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:07.022957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:07.022957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:07.023163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:07.023438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:07.023843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:07.023843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:07.023843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:07.023843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.023843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:07.043343 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:07.043786 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:07.043893 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:07.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:07.044480 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:07.044775 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:07.045023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.045715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:07.045715 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:07.045715 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:07.045715 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:07.045715 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:07.045715 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:07.045715 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.045715 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:07.046398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.046398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:07.046398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:07.046398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:07.046398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.046398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:07.046398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:07.046398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.046398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:07.046398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:07.046398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:07.046398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.046653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:07.046653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:07.046653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:07.046653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:07.046653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:07.046653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:07.046653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:07.046653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:07.046653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:07.046653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:07.046653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:07.046653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:07.046653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:07.046896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.046896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:07.047342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.047342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:07.047342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:07.047342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:07.047342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:07.047342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:07.047342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.047342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:07.047342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:07.047342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.047342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:07.047342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:07.047342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.047342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:07.047342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:07.047342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:07.047660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:07.047940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:07.047940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.047940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:07.067886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:07.068177 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:07.068267 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:07.071096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:07.097020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:07.097288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.097945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:07.097945 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:07.097945 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:07.097945 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:07.097945 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:07.097945 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:07.097945 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.097945 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:07.098667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.098667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:07.098667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:07.098667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:07.098667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.098667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:07.098667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:07.098667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.098667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:07.098667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:07.098667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:07.098667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.098889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:07.098889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:07.098889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:07.098889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:07.098889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:07.098889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:07.098889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:07.098889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:07.098889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:07.098889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:07.098889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:07.098889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:07.098889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:07.099101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.099101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:07.099356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.099356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:07.099356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:07.099356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:07.099356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:07.099356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:07.099356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.099356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:07.099356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:07.099356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.099356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:07.099356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:07.099356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.099356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:07.099356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:07.099356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:07.099655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:07.099954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:07.099954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:07.099954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:11.932034 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:11.932508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:11.932508 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:11.934913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:11.934913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:11.934913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:11.934913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:11.934913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:11.935144 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:11.935144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:11.935294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:11.935294 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:11.935294 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:11.935294 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:11.935294 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:11.935294 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:11.935294 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:11.935294 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:11.936324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:11.936324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:11.936324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:11.936324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:11.936324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:11.936324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:11.936324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:11.936324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:11.936324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:11.936324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:11.936324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:11.936324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:11.936724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:11.936724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:11.936724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:11.936724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:11.936724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:11.936724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:11.936724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:11.936724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:11.936724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:11.936724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:11.936724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:11.936724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:11.936724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:11.937033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:11.937033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:11.937848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:11.937848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:11.937848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:11.937848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:11.937848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:11.937848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:11.937848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:11.937848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:11.937848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:11.937848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:11.937848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:11.937848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:11.937848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:11.937848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:11.937848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:11.937848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:11.938201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:11.938754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:11.938754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:11.938754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:11.999440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:11.999935 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:12.000113 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:12.000277 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:45:12.000777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:12.001394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.001594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:12.001594 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:12.001594 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:12.001594 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:12.001594 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:12.001594 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:12.001594 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.001594 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:12.002296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.002296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:12.002296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:12.002296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:12.002296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.002296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:12.002296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:12.002296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:12.002296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.002296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:12.002296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:12.002296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:12.002296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:12.002296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:12.002296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:12.002678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:12.002678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:12.002678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:12.002678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:12.002678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:12.002678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:12.002678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:12.002678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.002678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:12.002935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.002935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:12.002935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:12.002935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:12.002935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.002935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:12.002935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:12.002935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:45:12.002935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:12.002935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:12.002935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.002935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:12.003170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:12.003170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:12.003170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.003170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:12.003170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:12.003170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:12.003170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:12.003170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:12.003170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:12.003170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:12.003170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:12.003170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:12.003170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:12.003170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:12.003379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:12.003379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:12.003379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.003379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:12.003677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.003677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:12.003677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:12.003677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:12.003677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:12.003677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:12.003677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.003677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:12.003677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:12.003677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.003677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:12.003677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:12.003677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.003677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:12.003677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:12.003677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:12.003999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:12.004331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:12.004331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:12.004331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:12.004331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:12.004331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:12.004331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.004331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:12.018184 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:12.018400 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:12.018527 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:12.018652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:12.019705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:12.019705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.019833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:12.019833 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:12.019833 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:12.019833 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:12.019833 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:12.019833 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:12.019833 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.019833 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:12.020427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.020427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:12.020427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:12.020427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:12.020427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.020427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:12.020427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:12.020427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:12.020427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.020427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:12.020427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:12.020427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:12.020427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:12.020427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:12.020427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:12.020735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:12.020735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:12.020735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:12.020735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:12.020735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:12.020735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:12.020735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:12.020735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.020735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:12.021030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.021030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:12.021030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:12.021030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:12.021030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.021030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:12.021030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:12.021030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:12.021030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:12.021030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.021030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:12.021285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:12.021285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:12.021285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.021285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:12.021285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:12.021285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:12.021285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:12.021285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:12.021285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:12.021285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:12.021285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:12.021285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:12.021285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:12.021285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:12.021596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:12.021596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:12.021596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.021596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.021833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:12.022200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:12.022602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:12.022602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:12.022602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.022602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:12.043804 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:12.044032 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:12.044120 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:12.044277 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:12.045256 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:12.045256 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:12.045256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.045382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:12.045382 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:12.045382 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:12.045382 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:12.045382 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:12.045382 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:12.045382 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.045382 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:12.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:12.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:12.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:12.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:12.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:12.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:12.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:12.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:12.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.046343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:12.046343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:12.046343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:12.046343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:12.046343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:12.046343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:12.046343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:12.046343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:12.046343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:12.046343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:12.046343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:12.046343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:12.046343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:12.046576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.046576 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:12.046728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.046728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:12.046728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:12.046728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:12.046728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:12.046728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:12.046728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.046728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:12.046728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:12.046728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.046728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:12.046728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:12.046728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.046728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:12.046728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:12.046728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:12.047060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:12.047386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:12.047386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.047386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:12.068248 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:12.068600 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:12.068695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:12.069512 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:12.111654 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:12.112502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.112763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:12.112763 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:12.112763 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:12.112763 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:12.112763 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:12.112763 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:12.112763 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.112763 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:12.113928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.044 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.113928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:12.113928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:12.113928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:12.113928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.113928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:12.113928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:12.113928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.113928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:12.113928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:12.113928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:12.113928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.114197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:12.114197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:12.114197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:12.114197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:12.114197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:12.114197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:12.114197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:12.114197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:12.114197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:12.114197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:12.114197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:12.114197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:12.114197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:12.114457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.114457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:12.115062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.115062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:12.115062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:12.115062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:12.115062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:12.115062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:12.115062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.115062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:12.115062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:12.115062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.115062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:12.115062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:12.115062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.115062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:12.115062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:12.115062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:12.115370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:12.115695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:12.115695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:12.115695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:16.932500 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:16.933003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:16.933059 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:16.935370 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:16.935370 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:16.935370 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:16.935370 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:16.935370 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:16.935634 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:16.935634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:16.935634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:16.935634 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:16.935634 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:16.935634 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:16.935634 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:16.935634 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:16.935634 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:16.935634 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:16.936349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:16.936349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:16.936349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:16.936349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:16.936349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:16.936349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:16.936349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:16.936349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:16.936349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:16.936349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:16.936349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:16.936349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:16.936789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:16.936789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:16.936789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:16.936789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:16.936789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:16.936789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:16.936789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:16.936789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:16.936789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:16.936789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:16.936789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:16.936789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:16.936789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:16.937158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:16.937158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:16.937228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:16.937228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:16.937228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:16.937228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:16.937228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:16.937228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:16.937228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:16.937228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:16.937228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:16.937228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:16.937228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:16.937228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:16.937228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:16.937228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:16.937228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:16.937228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:16.937734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:16.938273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:16.938273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:16.938273 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:16.999968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:17.000438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:17.000652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:17.000816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:45:17.001886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:17.001886 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.002045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:17.002045 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:17.002045 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:17.002045 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:17.002045 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:17.002045 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:17.002045 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.002045 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:17.002899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.002899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:17.002899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:17.002899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:17.002899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.002899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:17.002899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:17.002899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:17.002899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.002899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:17.002899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:17.002899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:17.002899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:17.002899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:17.002899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:17.003377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:17.003377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:17.003377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:17.003377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:17.003377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:17.003377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:17.003377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:17.003377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.003377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:17.003759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.003759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:17.003759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:17.003759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:17.003759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.003759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:17.003759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:17.003759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:45:17.003759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:17.003759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:17.003759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.003759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:17.004149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:17.004149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:17.004149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.004149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:17.004149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:17.004149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:17.004149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:17.004149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:17.004149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:17.004149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:17.004149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:17.004149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:17.004149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:17.004149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:17.004624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:17.004624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:17.004624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.004624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:17.004900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.004900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:17.004900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:17.004900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:17.004900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:17.004900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:17.004900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.004900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:17.004900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:17.004900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.004900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:17.004900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:17.004900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.004900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:17.004900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:17.004900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:17.005482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:17.006067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:17.006067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:17.006067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:17.006067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:17.006067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:17.006067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.006067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:17.018643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:17.018959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:17.019095 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:17.019149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:17.020158 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:17.020158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.020263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:17.020263 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:17.020263 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:17.020263 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:17.020263 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:17.020263 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:17.020263 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.020263 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:17.020879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.020879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:17.020879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:17.020879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:17.020879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.020879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:17.020879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:17.020879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:17.020879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.020879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:17.020879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:17.020879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:17.020879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:17.020879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:17.020879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:17.021292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:17.021292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:17.021292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:17.021292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:17.021292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:17.021292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:17.021292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:17.021292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.021292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:17.021489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.021489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:17.021489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:17.021489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:17.021489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.021489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:17.021489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:17.021489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:17.021489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:17.021489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.021489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:17.021698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:17.021698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:17.021698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.021698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:17.021698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:17.021698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:17.021698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:17.021698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:17.021698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:17.021698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:17.021698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:17.021698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:17.021698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:17.021698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:17.021998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:17.021998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:17.021998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.021998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.022156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:17.022518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:17.023012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:17.023012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:17.023012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.023012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:17.044353 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:17.044693 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:17.044808 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:17.044955 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:17.045270 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:17.045572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:17.045814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.046263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:17.046263 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:17.046263 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:17.046263 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:17.046263 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:17.046263 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:17.046263 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.046263 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:17.046861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.046861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:17.046861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:17.046861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:17.046861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.046861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:17.046861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:17.046861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.046861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:17.046861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:17.046861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:17.046861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.047255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:17.047255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:17.047255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:17.047255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:17.047255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:17.047255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:17.047255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:17.047255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:17.047255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:17.047255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:17.047255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:17.047255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:17.047255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:17.047694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.047694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:17.047694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.047694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:17.047694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:17.047694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:17.047694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:17.047694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:17.047694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.047694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:17.047694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:17.047694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.047694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:17.047694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:17.047694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:17.048149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:17.048705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:17.048705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:17.048705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:17.048705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:17.048705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.048705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:17.068845 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:17.069195 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:17.069300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:17.069936 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:17.100703 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:17.100703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.100869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:17.100869 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:17.100869 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:17.100869 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:17.100869 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:17.100869 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:17.100869 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.100869 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:17.101718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.101718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:17.101718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:17.101718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:17.101718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.101718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:17.101718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:17.101718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.101718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:17.101718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:17.101718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:17.101718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.102148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:17.102148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:17.102148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:17.102148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:17.102148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:17.102148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:17.102148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:17.102148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:17.102148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:17.102148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:17.102148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:17.102148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:17.102148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:17.102505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.102505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:17.102505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.102505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:17.102505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:17.102505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:17.102505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:17.102505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:17.102505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.102505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:17.102505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:17.102505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.102505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:17.102505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:17.102505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:17.102912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:17.103371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:17.103371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:17.103371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:17.103371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:17.103371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:17.103371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:21.933103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:21.933754 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:21.933754 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:21.934277 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:21.934430 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:21.934633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:21.934719 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:21.934822 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:21.934887 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:21.934978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:21.935364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:21.935364 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:21.935364 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:21.935364 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:21.935364 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:21.935364 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:21.935364 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:21.935364 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:21.936028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:21.936028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:21.936028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:21.936028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:21.936028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:21.936028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:21.936028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:21.936028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:21.936028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:21.936028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:21.936028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:21.936028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:21.936397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:21.936397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:21.936397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:21.936397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:21.936397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:21.936397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:21.936397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:21.936397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:21.936397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:21.936397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:21.936397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:21.936397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:21.936397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:21.936715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:21.936715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:21.936775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:21.936775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:21.936775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:21.936775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:21.936775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:21.936775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:21.936775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:21.936775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:21.936775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:21.936775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:21.936775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:21.936775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:21.936775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:21.936775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:21.936775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:21.936775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:21.937050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:21.937424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:21.937424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:21.937424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:22.000438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:22.001004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:22.001121 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:22.001274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:45:22.001643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:22.001815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.002246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:22.002246 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:22.002246 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:22.002246 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:22.002246 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:22.002246 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:22.002246 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.002246 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:22.003090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.003090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:22.003090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:22.003090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:22.003090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.003090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:22.003090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:22.003090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:22.003090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.003090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:22.003090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:22.003090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:22.003090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:22.003090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:22.003090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:22.003388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:22.003388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:22.003388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:22.003388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:22.003388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:22.003388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:22.003388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:22.003388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.003388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:22.003998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.003998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:22.003998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:22.003998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:22.003998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.003998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:22.003998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:22.003998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:45:22.003998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:22.003998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:22.003998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.003998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:22.004235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:22.004235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:22.004235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.004235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:22.004235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:22.004235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:22.004235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:22.004235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:22.004235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:22.004235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:22.004235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:22.004235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:22.004235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:22.004235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:22.004456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:22.004456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:22.004456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.004456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:22.005156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.005156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:22.005156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:22.005156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:22.005156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:22.005156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:22.005156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.005156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:22.005156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:22.005156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.005156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:22.005156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:22.005156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.005156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:22.005156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:22.005156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:22.006043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:22.006374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:22.006374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:22.006374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:22.006374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:22.006374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:22.006374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.006374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:22.019117 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:22.019526 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:22.019711 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:22.019832 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:22.020181 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:22.020392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.020850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:22.020850 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:22.020850 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:22.020850 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:22.020850 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:22.020850 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:22.020850 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.020850 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:22.021387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.021387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:22.021387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:22.021387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:22.021387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.021387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:22.021387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:22.021387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:22.021387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.021387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:22.021387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:22.021387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:22.021387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:22.021387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:22.021387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:22.021730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:22.021730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:22.021730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:22.021730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:22.021730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:22.021730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:22.021730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:22.021730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.021730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:22.022083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.022083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:22.022083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:22.022083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:22.022083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.022083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:22.022083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:22.022083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:22.022083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:22.022083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.022083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:22.022330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:22.022330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:22.022330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.022330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:22.022330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:22.022330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:22.022330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:22.022330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:22.022330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:22.022330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:22.022330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:22.022330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:22.022330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:22.022330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:22.022625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:22.022625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:22.022625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.022625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.023022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:22.023695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:22.024009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:22.024009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:22.024009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.024009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:22.044810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:22.045081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:22.045186 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:22.045327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:22.045844 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:22.046135 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:22.046430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.046997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:22.046997 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:22.046997 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:22.046997 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:22.046997 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:22.046997 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:22.046997 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.046997 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:22.047835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.047835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:22.047835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:22.047835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:22.047835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.047835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:22.047835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:22.047835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.047835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:22.047835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:22.047835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:22.047835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.048058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:22.048058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:22.048058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:22.048058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:22.048058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:22.048058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:22.048058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:22.048058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:22.048058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:22.048058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:22.048058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:22.048058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:22.048058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:22.048258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.048258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:22.048729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.048729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:22.048729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:22.048729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:22.048729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:22.048729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:22.048729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.048729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:22.048729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:22.048729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.048729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:22.048729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:22.048729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.048729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:22.048729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:22.048729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:22.049043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:22.049333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:22.049333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.049333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:22.069318 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:22.069706 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:22.069860 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:22.070642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:22.082883 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=240 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:45:22.102984 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:22.103172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.103689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:22.103689 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:22.103689 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:22.103689 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:22.103689 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:22.103689 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:22.103689 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.103689 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:22.104377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.104377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:22.104377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:22.104377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:22.104377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.104377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:22.104377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:22.104377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.104377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:22.104377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:22.104377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:22.104377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.104757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:22.104757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:22.104757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:22.104757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:22.104757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:22.104757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:22.104757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:22.104757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:22.104757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:22.104757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:22.104757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:22.104757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:22.104757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:22.105174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.105174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:22.105740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.105740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:22.105740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:22.105740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:22.105740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:22.105740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:22.105740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.105740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:22.105740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:22.105740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.105740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:22.105740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:22.105740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.105740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:22.105740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:22.105740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:22.106779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:22.107233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:22.107233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:22.107233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:26.933511 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:26.934050 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:26.934050 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:26.934572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:26.934714 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:26.934826 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:26.934926 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:26.935021 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:26.935110 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:26.935206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:26.935662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:26.935662 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:26.935662 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:26.935662 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:26.935662 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:26.935662 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:26.935662 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:26.935662 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:26.936634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:26.936634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:26.936634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:26.936634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:26.936634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:26.936634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:26.936634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:26.936634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:26.936634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:26.936634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:26.936634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:26.936634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:26.936911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:26.936911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:26.936911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:26.936911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:26.936911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:26.936911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:26.936911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:26.936911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:26.936911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:26.936911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:26.936911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:26.936911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:26.936911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:26.937114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:26.937114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:26.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:26.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:26.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:26.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:26.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:26.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:26.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:26.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:26.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:26.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:26.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:26.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:26.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:26.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:26.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:26.937744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:26.938205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:26.938695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:26.938695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:26.938695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:27.000849 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:27.001117 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:27.001210 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:27.001315 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:45:27.001650 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:27.001806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.002196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:27.002196 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:27.002196 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:27.002196 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:27.002196 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:27.002196 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:27.002196 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.002196 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:27.002770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.002770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:27.002770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:27.002770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:27.002770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.002770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:27.002770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:27.002770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:27.002770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.002770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:27.002770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:27.002770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:27.002770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:27.002770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:27.002770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:27.003064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:27.003064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:27.003064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:27.003064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:27.003064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:27.003064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:27.003064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:27.003064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.003064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:27.003384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.003384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:27.003384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:27.003384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:27.003384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.003384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:27.003384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:27.003384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:45:27.003384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:27.003384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:27.003384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.003384 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:27.003643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:27.003643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:27.003643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.003643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:27.003643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:27.003643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:27.003643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:27.003643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:27.003643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:27.003643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:27.003643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:27.003643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:27.003643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:27.003643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:27.003859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:27.003859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:27.003859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.003859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:27.004073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.004073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:27.004073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:27.004073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:27.004073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:27.004073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:27.004073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.004073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:27.004073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:27.004073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.004073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:27.004073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:27.004073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.004073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:27.004073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:27.004073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:27.004336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:27.004655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:27.004655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:27.004655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:27.004655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:27.004655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:27.004655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.004655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:27.019530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:27.019800 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:27.019897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:27.020002 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:27.020312 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:27.020560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.021004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:27.021004 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:27.021004 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:27.021004 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:27.021004 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:27.021004 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:27.021004 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.021004 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:27.021553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.021553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:27.021553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:27.021553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:27.021553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.021553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:27.021553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:27.021553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:27.021553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.021553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:27.021553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:27.021553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:27.021553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:27.021553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:27.021553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:27.021858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:27.021858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:27.021858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:27.021858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:27.021858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:27.021858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:27.021858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:27.021858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.021858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:27.022138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.022138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:27.022138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:27.022138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:27.022138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.022138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:27.022138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:27.022138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:27.022138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:27.022138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.022138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:27.022350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:27.022350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:27.022350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.022350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:27.022350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:27.022350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:27.022350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:27.022350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:27.022350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:27.022350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:27.022350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:27.022350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:27.022350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:27.022350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:27.022613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:27.022613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:27.022613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.022613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.022914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:27.023217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:27.023525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:27.023525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:27.023525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.023525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:27.045196 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:27.045576 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:27.045675 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:27.045827 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:27.046232 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:27.046490 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:27.046782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.047264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:27.047264 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:27.047264 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:27.047264 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:27.047264 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:27.047264 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:27.047264 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.047264 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:27.048377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.048377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:27.048377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:27.048377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:27.048377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.048377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:27.048377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:27.048377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.048377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:27.048377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:27.048377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:27.048377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.048628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:27.048628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:27.048628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:27.048628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:27.048628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:27.048628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:27.048628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:27.048628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:27.048628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:27.048628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:27.048628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:27.048628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:27.048628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:27.048839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.048839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:27.049067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.049067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:27.049067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:27.049067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:27.049067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:27.049067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:27.049067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.049067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:27.049067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:27.049067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.049067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:27.049067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:27.049067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.049067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:27.049067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:27.049067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:27.049351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:27.049687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:27.049687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.049687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:27.069747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:27.070080 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:27.070165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:27.070803 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:27.100056 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:27.100272 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.100831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:27.100831 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:27.100831 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:27.100831 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:27.100831 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:27.100831 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:27.100831 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.100831 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:27.101518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.101518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:27.101518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:27.101518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:27.101518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.101518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:27.101518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:27.101518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.101518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:27.101518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:27.101518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:27.101518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.101777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:27.101777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:27.101777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:27.101777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:27.101777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:27.101777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:27.101777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:27.101777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:27.101777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:27.101777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:27.101777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:27.101777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:27.101777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:27.101984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.101984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:27.102250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.102250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:27.102250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:27.102250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:27.102250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:27.102250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:27.102250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.102250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:27.102250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:27.102250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.102250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:27.102250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:27.102250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.102250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:27.102250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:27.102250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:27.102526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:27.102865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:27.102865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:27.102865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:31.933990 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:31.934512 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:31.934512 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:31.935026 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:31.935165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:31.935271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:31.935389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:31.935520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:31.935635 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:31.935737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:31.936135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:31.936135 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:31.936135 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:31.936135 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:31.936135 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:31.936135 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:31.936135 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:31.936135 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:31.936791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:31.936791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:31.936791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:31.936791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:31.936791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:31.936791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:31.936791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:31.936791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:31.936791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:31.936791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:31.936791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:31.936791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:31.937088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:31.937088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:31.937088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:31.937088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:31.937088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:31.937088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:31.937088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:31.937088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:31.937088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:31.937088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:31.937088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:31.937088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:31.937088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:31.937349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:31.937349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:31.937711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:31.937711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:31.937711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:31.937711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:31.937711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:31.937711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:31.937711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:31.937711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:31.937711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:31.937711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:31.937711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:31.937711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:31.937711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:31.937711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:31.937711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:31.937711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:31.938660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:31.939021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:31.939021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:31.939021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:32.001308 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:32.001476 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:32.001625 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:32.001711 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:45:32.002010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:32.002188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.002651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:32.002651 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:32.002651 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:32.002651 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:32.002651 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:32.002651 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:32.002651 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.002651 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:32.003224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.003224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:32.003224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:32.003224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:32.003224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.003224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:32.003224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:32.003224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:32.003224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.003224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:32.003224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:32.003224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:32.003224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:32.003224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:32.003224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:32.003534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:32.003534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:32.003534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:32.003534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:32.003534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:32.003534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:32.003534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:32.003534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.003534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:32.003942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.003942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:32.003942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:32.003942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:32.003942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.003942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:32.003942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:32.003942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:45:32.003942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:32.003942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:32.003942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.003942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:32.004748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:32.004748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:32.004748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.004748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:32.004748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:32.004748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:32.004748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:32.004748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:32.004748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:32.004748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:32.004748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:32.004748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:32.004748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:32.004748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:32.005022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:32.005022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:32.005022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.005022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:32.005022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.005022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:32.005022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:32.005022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:32.005022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:32.005022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:32.005022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.005022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:32.005022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:32.005022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:32.005270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:32.005620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:32.005620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:32.005620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:32.005620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:32.005620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:32.005620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:32.005620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:32.005620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:32.005620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:32.005620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:32.005620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.005620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:32.020064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:32.020188 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:32.020347 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:32.020405 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:32.020692 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:32.020879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.021294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:32.021294 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:32.021294 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:32.021294 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:32.021294 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:32.021294 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:32.021294 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.021294 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:32.021846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.021846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:32.021846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:32.021846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:32.021846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.021846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:32.021846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:32.021846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:32.021846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.021846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:32.021846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:32.021846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:32.021846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:32.021846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:32.021846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:32.022126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:32.022126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:32.022126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:32.022126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:32.022126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:32.022126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:32.022126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:32.022126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.022126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:32.022504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.022504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:32.022504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:32.022504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:32.022504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.022504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:32.022504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:32.022504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:32.022504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:32.022504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.022504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:32.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:32.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:32.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:32.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:32.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:32.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:32.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:32.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:32.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:32.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:32.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:32.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:32.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:32.023505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:32.023505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:32.023505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.023505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:32.023505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.023505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:32.023505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:32.023505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:32.023505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:32.023505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:32.023505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.023505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:32.023505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:32.023505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:32.023774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:32.024095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:32.024095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:32.024095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:32.024095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:32.024095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:32.024095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:32.024095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:32.024095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:32.024095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:32.024095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.024095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:32.045665 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:32.045841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:32.045908 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:32.046062 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:32.046354 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:32.046577 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:32.046770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.047147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:32.047147 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:32.047147 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:32.047147 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:32.047147 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:32.047147 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:32.047147 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.047147 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:32.047778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.047778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:32.047778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:32.047778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:32.047778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.047778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:32.047778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:32.047778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.047778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:32.047778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:32.047778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:32.047778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.048021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:32.048021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:32.048021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:32.048021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:32.048021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:32.048021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:32.048021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:32.048021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:32.048021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:32.048021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:32.048021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:32.048021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:32.048021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:32.048278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.048278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:32.048643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.048643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:32.048643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:32.048643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:32.048643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:32.048643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:32.048643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.048643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:32.048643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:32.048643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.048643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:32.048643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:32.048643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.048643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:32.048643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:32.048643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:32.049424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:32.049802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:32.049802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.049802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:32.070203 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:32.070382 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:32.070484 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:32.071060 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:32.098727 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:32.098727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.098727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:32.098727 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:32.098727 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:32.098727 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:32.098727 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:32.098727 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:32.098727 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.098727 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:32.099195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.099195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:32.099195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:32.099195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:32.099195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.099195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:32.099195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:32.099195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.099195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:32.099195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:32.099195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:32.099195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.099619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:32.099619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:32.099619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:32.099619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:32.099619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:32.099619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:32.099619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:32.099619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:32.099619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:32.099619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:32.099619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:32.099619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:32.099619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:32.099852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.099852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:32.099894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.099894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:32.099894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:32.099894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:32.099894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:32.099894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:32.099894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.099894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:32.099894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:32.099894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.099894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:32.099894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:32.099894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.099894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:32.099894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:32.099894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:32.100143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:32.100429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:32.100429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:32.100429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:36.934329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:36.934964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:36.934964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:36.935748 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:36.935938 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:36.936066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:36.936169 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:36.936281 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:36.936385 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:36.936515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:36.937002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:36.937002 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:36.937002 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:36.937002 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:36.937002 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:36.937002 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:36.937002 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:36.937002 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:36.937845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:36.937845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:36.937845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:36.937845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:36.937845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:36.937845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:36.937845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:36.937845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:36.937845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:36.937845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:36.937845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:36.937845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:36.938143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:36.938143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:36.938143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:36.938143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:36.938143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:36.938143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:36.938143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:36.938143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:36.938143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:36.938143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:36.938143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:36.938143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:36.938143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:36.938478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:36.938478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:36.938729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:36.938729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:36.938729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:36.938729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:36.938729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:36.938729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:36.938729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:36.938729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:36.938729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:36.938729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:36.938729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:36.938729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:36.938729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:36.938729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:36.938729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:36.938729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:36.939596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:36.940476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:36.940476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:36.940476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:37.001683 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:37.002040 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:37.002143 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:37.002236 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:45:37.002826 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:37.003047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.003525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:37.003525 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:37.003525 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:37.003525 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:37.003525 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:37.003525 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:37.003525 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.003525 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:37.004198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.004198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:37.004198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:37.004198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:37.004198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.004198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:37.004198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:37.004198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:37.004198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.004198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:37.004198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:37.004198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:37.004198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:37.004198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:37.004198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:37.004581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:37.004581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:37.004581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:37.004581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:37.004581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:37.004581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:37.004581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:37.004581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.004581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:37.004993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.004993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:37.004993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:37.004993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:37.004993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.004993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:37.004993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:37.004993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:45:37.004993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:37.004993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:37.004993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.004993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:37.005707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:37.005707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:37.005707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.005707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:37.005707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:37.005707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:37.005707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:37.005707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:37.005707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:37.005707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:37.005707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:37.005707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:37.005707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:37.005707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:37.006322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:37.006322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:37.006322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.006322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:37.006322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.006322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:37.006322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:37.006322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:37.006322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:37.006322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:37.006322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.006322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:37.006322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:37.006322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:37.006682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:37.007112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:37.007112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:37.007112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:37.007112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:37.007112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:37.007112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:37.007112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:37.007112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:37.007112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:37.007112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:37.007112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.007112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:37.020333 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:37.020799 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:37.020896 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:37.020980 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:37.021468 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:37.021716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.022159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:37.022159 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:37.022159 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:37.022159 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:37.022159 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:37.022159 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:37.022159 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.022159 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:37.022832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.022832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:37.022832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:37.022832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:37.022832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.022832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:37.022832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:37.022832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:37.022832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.022832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:37.022832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:37.022832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:37.022832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:37.022832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:37.022832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:37.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:37.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:37.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:37.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:37.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:37.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:37.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:37.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.023188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:37.023397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.023397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:37.023397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:37.023397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:37.023397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.023397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:37.023397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:37.023397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:37.023397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:37.023397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.023397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:37.023704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:37.023704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:37.023704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.023704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:37.023704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:37.023704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:37.023704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:37.023704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:37.023704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:37.023704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:37.023704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:37.023704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:37.023704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:37.023704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:37.024022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:37.024022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:37.024022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.024022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.024605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:37.025406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:37.026413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:37.026413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:37.026413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.026413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:37.046009 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:37.046266 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:37.046387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:37.046625 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:37.047061 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:37.047279 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:37.047520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.048040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:37.048040 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:37.048040 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:37.048040 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:37.048040 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:37.048040 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:37.048040 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.048040 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:37.049038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.049038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:37.049038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:37.049038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:37.049038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.049038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:37.049038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:37.049038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.049038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:37.049038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:37.049038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:37.049038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.049347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:37.049347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:37.049347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:37.049347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:37.049347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:37.049347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:37.049347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:37.049347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:37.049347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:37.049347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:37.049347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:37.049347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:37.049347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:37.049682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.049682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:37.050119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.050119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:37.050119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:37.050119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:37.050119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:37.050119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:37.050119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.050119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:37.050119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:37.050119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.050119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:37.050119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:37.050119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.050119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:37.050119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:37.050119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:37.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:37.051029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:37.051029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.051029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:37.070559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:37.070807 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:37.070936 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:37.071741 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:37.112523 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:37.112746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.113304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:37.113304 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:37.113304 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:37.113304 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:37.113304 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:37.113304 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:37.113304 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.113304 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:37.114086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.042 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.114086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:37.114086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:37.114086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:37.114086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.114086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:37.114086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:37.114086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.114086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:37.114086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:37.114086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:37.114086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.114322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:37.114322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:37.114322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:37.114322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:37.114322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:37.114322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:37.114322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:37.114322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:37.114322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:37.114322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:37.114322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:37.114322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:37.114322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:37.114554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.114554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:37.114896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.114896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:37.114896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:37.114896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:37.114896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:37.114896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:37.114896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.114896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:37.114896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:37.114896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.114896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:37.114896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:37.114896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.114896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:37.114896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:37.114896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:37.115496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:37.116130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:37.116130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:37.116130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:41.934789 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:41.935223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:41.935223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:41.936104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:41.936156 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:41.936284 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:41.936395 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:41.936530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:41.936678 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:41.936784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:41.937257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:41.937257 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:41.937257 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:41.937257 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:41.937257 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:41.937257 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:41.937257 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:41.937257 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:41.938194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:41.938194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:41.938194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:41.938194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:41.938194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:41.938194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:41.938194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:41.938194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:41.938194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:41.938194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:41.938194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:41.938194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:41.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:41.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:41.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:41.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:41.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:41.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:41.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:41.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:41.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:41.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:41.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:41.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:41.938497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:41.938774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:41.938774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:41.939332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:41.939332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:41.939332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:41.939332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:41.939332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:41.939332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:41.939332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:41.939332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:41.939332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:41.939332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:41.939332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:41.939332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:41.939332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:41.939332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:41.939332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:41.939332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:41.939711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:41.940127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:41.940127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:41.940127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:42.002068 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:42.002327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:42.002419 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:42.002536 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:45:42.003014 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:42.003182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.004232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:42.004232 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:42.004232 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:42.004232 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:42.004232 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:42.004232 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:42.004232 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.004232 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:42.004895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.004895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:42.004895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:42.004895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:42.004895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.004895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:42.004895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:42.004895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:42.004895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.004895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:42.004895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:42.004895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:42.004895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:42.004895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:42.004895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:42.005200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:42.005200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:42.005200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:42.005200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:42.005200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:42.005200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:42.005200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:42.005200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.005200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:42.005493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.005493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:42.005493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:42.005493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:42.005493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.005493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:42.005493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:42.005493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:45:42.005493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:42.005493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:42.005493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.005493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:42.005748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:42.005748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:42.005748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.005748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:42.005748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:42.005748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:42.005748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:42.005748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:42.005748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:42.005748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:42.005748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:42.005748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:42.005748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:42.005748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:42.005997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:42.005997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:42.005997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.005997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:42.006251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.006251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:42.006251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:42.006251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:42.006251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:42.006251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:42.006251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.006251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:42.006251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:42.006251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.006251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:42.006251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:42.006251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.006251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:42.006251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:42.006251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:42.006586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:42.006924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:42.006924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:42.006924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:42.006924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:42.006924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:42.006924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.006924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:42.020991 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:42.021278 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:42.021709 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:42.021709 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:42.022016 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:42.022220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.022717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:42.022717 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:42.022717 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:42.022717 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:42.022717 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:42.022717 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:42.022717 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.022717 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:42.023602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.023602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:42.023602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:42.023602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:42.023602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.023602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:42.023602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:42.023602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:42.023602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.023602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:42.023602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:42.023602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:42.023602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:42.023602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:42.023602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:42.023885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:42.023885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:42.023885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:42.023885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:42.023885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:42.023885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:42.023885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:42.023885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.023885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:42.024600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.024600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:42.024600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:42.024600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:42.024600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.024600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:42.024600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:42.024600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:42.024600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:42.024600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.024600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:42.024837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:42.024837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:42.024837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.024837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:42.024837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:42.024837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:42.024837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:42.024837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:42.024837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:42.024837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:42.024837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:42.024837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:42.024837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:42.024837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:42.025093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:42.025093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:42.025093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.025093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.025897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:42.026222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:42.026550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:42.026550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:42.026550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.026550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:42.046395 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:42.046706 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:42.046827 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:42.047006 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:42.047440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:42.047681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:42.047844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.048349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:42.048349 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:42.048349 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:42.048349 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:42.048349 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:42.048349 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:42.048349 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.048349 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:42.049355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.049355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:42.049355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:42.049355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:42.049355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.049355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:42.049355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:42.049355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.049355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:42.049355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:42.049355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:42.049355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.049632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:42.049632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:42.049632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:42.049632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:42.049632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:42.049632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:42.049632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:42.049632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:42.049632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:42.049632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:42.049632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:42.049632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:42.049632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:42.049867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.049867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:42.050403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.050403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:42.050403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:42.050403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:42.050403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:42.050403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:42.050403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.050403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:42.050403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:42.050403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.050403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:42.050403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:42.050403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.050403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:42.050403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:42.050403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:42.050775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:42.051096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:42.051096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.051096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:42.070853 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:42.071085 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:42.071203 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:42.072036 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:42.103757 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:42.104054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.104611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:42.104611 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:42.104611 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:42.104611 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:42.104611 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:42.104611 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:42.104611 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.104611 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:42.105307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.105307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:42.105307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:42.105307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:42.105307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.105307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:42.105307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:42.105307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.105307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:42.105307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:42.105307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:42.105307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.105612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:42.105612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:42.105612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:42.105612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:42.105612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:42.105612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:42.105612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:42.105612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:42.105612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:42.105612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:42.105612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:42.105612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:42.105612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:42.105869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.105869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:42.106021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.106021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:42.106021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:42.106021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:42.106021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:42.106021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:42.106021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.106021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:42.106021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:42.106021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.106021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:42.106021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:42.106021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.106021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:42.106021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:42.106021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:42.106605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:42.106955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:42.106955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:42.106955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:46.935215 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:46.935762 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:46.935762 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:46.936451 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:46.936642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:46.936798 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:46.936941 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:46.937079 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:46.937225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:46.937359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:46.937946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:46.937946 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:46.937946 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:46.937946 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:46.937946 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:46.937946 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:46.937946 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:46.937946 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:46.938899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:46.938899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:46.938899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:46.938899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:46.938899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:46.938899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:46.938899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:46.938899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:46.938899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:46.938899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:46.938899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:46.938899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:46.939244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:46.939244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:46.939244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:46.939244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:46.939244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:46.939244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:46.939244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:46.939244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:46.939244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:46.939244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:46.939244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:46.939244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:46.939244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:46.939648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:46.939648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:46.939979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:46.939979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:46.939979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:46.939979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:46.939979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:46.939979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:46.939979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:46.939979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:46.939979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:46.939979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:46.939979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:46.939979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:46.939979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:46.939979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:46.939979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:46.939979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:46.940434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:46.940943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:46.940943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:46.940943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:47.002937 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:47.003084 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:47.003139 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:47.003255 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:45:47.003753 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:47.003958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.004391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:47.004391 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:47.004391 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:47.004391 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:47.004391 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:47.004391 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:47.004391 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.004391 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:47.005012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.005012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:47.005012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:47.005012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:47.005012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.005012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:47.005012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:47.005012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:47.005012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.005012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:47.005012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:47.005012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:47.005012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:47.005012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:47.005012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:47.005330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:47.005330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:47.005330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:47.005330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:47.005330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:47.005330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:47.005330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:47.005330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.005330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:47.005647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.005647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:47.005647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:47.005647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:47.005647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.005647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:47.005647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:47.005647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:45:47.005647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:47.005647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:47.005647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.005647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:47.005972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:47.005972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:47.005972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.005972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:47.005972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:47.005972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:47.005972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:47.005972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:47.005972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:47.005972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:47.005972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:47.005972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:47.005972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:47.005972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:47.006325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:47.006325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:47.006325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.006325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:47.006325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.006325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:47.006325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:47.006325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:47.006325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:47.006325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:47.006325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.006325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:47.006325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:47.006325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:47.006630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:47.006952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:47.006952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:47.006952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:47.006952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:47.006952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:47.006952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:47.006952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:47.006952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:47.006952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:47.006952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:47.006952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.006952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:47.021437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:47.021711 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:47.021806 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:47.022423 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:47.023013 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:47.023319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.023982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:47.023982 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:47.023982 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:47.023982 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:47.023982 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:47.023982 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:47.023982 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.023982 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:47.255479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.255479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:47.255479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:47.255479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:47.255479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.255479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:47.255479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:47.255479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:47.255479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.255479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:47.255479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:47.255479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:47.255479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:47.255479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:47.255479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:47.255897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:47.255897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:47.255897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:47.255897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:47.255897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:47.255897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:47.255897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:47.255897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.255897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:47.255897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:47.256509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.234 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.256509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:47.256509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:47.256509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:47.256509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.256509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:47.256509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:47.256509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:47.256509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:47.256509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.256509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:47.256820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:47.256820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:47.256820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.256820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:47.256820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:47.256820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:47.256820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:47.256820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:47.256820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:47.256820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:47.256820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:47.256820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:47.256820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:47.256820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:47.257055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:47.257055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:47.257055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.257055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:47.257055 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.257718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:47.258092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:47.258580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:47.258580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:47.258580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.258580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:47.258580 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:47.258580 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:47.258580 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:47.259090 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:47.259396 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:47.259682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.260142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:47.260142 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:47.260142 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:47.260142 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:47.260142 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:47.260142 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:47.260142 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.260142 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:47.260809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.260809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:47.260809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:47.260809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:47.260809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.260809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:47.260809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:47.260809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.260809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:47.260809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:47.260809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:47.260809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.261016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:47.261016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:47.261016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:47.261016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:47.261016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:47.261016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:47.261016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:47.261016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:47.261016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:47.261016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:47.261016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:47.261016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:47.261016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:47.261236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.261236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:47.261698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.261698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:47.261698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:47.261698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:47.261698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:47.261698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:47.261698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.261698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:47.261698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:47.261698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.261698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:47.261698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:47.261698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.261698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:47.261698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:47.261698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:47.262024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:47.262623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:47.262623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.262623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:47.262623 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:47.262623 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:47.262813 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:47.297520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:47.297734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.298351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:47.298351 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:47.298351 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:47.298351 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:47.298351 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:47.298351 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:47.298351 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.298351 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:47.299437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.299437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:47.299437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:47.299437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:47.299437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.299437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:47.299437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:47.299437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.299437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:47.299437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:47.299437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:47.299437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.299853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:47.299853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:47.299853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:47.299853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:47.299853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:47.299853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:47.299853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:47.299853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:47.299853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:47.299853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:47.299853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:47.299853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:47.299853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:47.300189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.300189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:47.300737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.300737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:47.300737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:47.300737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:47.300737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:47.300737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:47.300737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.300737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:47.300737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:47.300737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.300737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:47.300737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:47.300737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.300737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:47.300737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:47.300737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:47.301016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:47.301343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:47.301343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:47.301343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:51.935622 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:51.936174 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:51.936174 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:51.937660 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:51.937833 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:51.937950 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:51.938049 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:51.938156 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:51.938258 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:51.938387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:51.938504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:51.938966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:51.938966 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:51.938966 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:51.938966 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:51.938966 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:51.938966 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:51.938966 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:51.938966 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:51.939847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:51.939847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:51.939847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:51.939847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:51.939847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:51.939847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:51.939847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:51.939847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:51.939847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:51.939847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:51.939847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:51.939847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:51.940491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:51.940491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:51.940491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:51.940491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:51.940491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:51.940491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:51.940491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:51.940491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:51.940491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:51.940491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:51.940491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:51.940491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:51.940491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:51.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:51.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:51.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:51.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:51.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:51.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:51.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:51.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:51.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:51.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:51.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:51.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:51.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:51.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:51.941693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:51.942406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:51.943468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:51.943468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:51.943468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:51.943468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:51.943468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:51.943468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:52.003319 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:52.003609 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:52.003701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:52.003798 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:45:52.004250 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:52.004476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.004986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:52.004986 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:52.004986 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:52.004986 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:52.004986 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:52.004986 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:52.004986 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.004986 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:52.005588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.005588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:52.005588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:52.005588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:52.005588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.005588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:52.005588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:52.005588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:52.005588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.005588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:52.005588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:52.005588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:52.005588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:52.005588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:52.005588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:52.005855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:52.005855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:52.005855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:52.005855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:52.005855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:52.005855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:52.005855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:52.005855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.005855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:52.006170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.006170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:52.006170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:52.006170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:52.006170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.006170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:52.006170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:52.006170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:45:52.006170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:52.006170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:52.006170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.006170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:52.006765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:52.006765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:52.006765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.006765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:52.006765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:52.006765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:52.006765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:52.006765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:52.006765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:52.006765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:52.006765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:52.006765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:52.006765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:52.006765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:52.007231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:52.007231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:52.007231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.007231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:52.007231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.007231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:52.007231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:52.007231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:52.007231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:52.007231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:52.007231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.007231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:52.007231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:52.007231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:52.007493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:52.007853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:52.007853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:52.007853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:52.007853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:52.007853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:52.007853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:52.007853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:52.007853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:52.007853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:52.007853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:52.007853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.007853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:52.021892 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:52.022161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:52.022287 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:52.022424 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:52.022781 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:52.022985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.023413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:52.023413 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:52.023413 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:52.023413 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:52.023413 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:52.023413 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:52.023413 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.023413 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:52.024006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.024006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:52.024006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:52.024006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:52.024006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.024006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:52.024006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:52.024006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:52.024006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.024006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:52.024006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:52.024006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:52.024006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:52.024006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:52.024006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:52.024323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:52.024323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:52.024323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:52.024323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:52.024323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:52.024323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:52.024323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:52.024323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.024323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:52.024832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.024832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:52.024832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:52.024832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:52.024832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.024832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:52.024832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:52.024832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:52.024832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:52.024832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.024832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:52.025100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:52.025100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:52.025100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.025100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:52.025100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:52.025100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:52.025100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:52.025100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:52.025100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:52.025100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:52.025100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:52.025100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:52.025100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:52.025100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:52.025363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:52.025363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:52.025363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.025363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.025867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:52.026223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:52.026621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:52.026621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:52.026621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.026621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:52.258715 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:52.258925 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:52.259041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:52.259184 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:52.260197 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:52.260197 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:52.260197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.261585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:52.261585 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:52.261585 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:52.261585 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:52.261585 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:52.261585 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:52.261585 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.261585 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:52.262342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.262342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:52.262342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:52.262342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:52.262342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.262342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:52.262342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:52.262342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.262342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:52.262342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:52.262342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:52.262342 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.262659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:52.262659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:52.262659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:52.262659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:52.262659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:52.262659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:52.262659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:52.262659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:52.262659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:52.262659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:52.262659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:52.262659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:52.262659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:52.262920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.262920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:52.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:52.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:52.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:52.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:52.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:52.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:52.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:52.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:52.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:52.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:52.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:52.263223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:52.263567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:52.263885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:52.263885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.263885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:52.263885 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:52.264163 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:52.264249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:52.265128 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:52.293373 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:52.294607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.294814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:52.294814 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:52.294814 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:52.294814 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:52.294814 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:52.294814 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:52.294814 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.294814 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:52.295485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.295485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:52.295485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:52.295485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:52.295485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.295485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:52.295485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:52.295485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.295485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:52.295485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:52.295485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:52.295485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.295770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:52.295770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:52.295770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:52.295770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:52.295770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:52.295770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:52.295770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:52.295770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:52.295770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:52.295770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:52.295770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:52.295770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:52.295770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:52.296030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.296030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:52.296268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.296268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:52.296268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:52.296268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:52.296268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:52.296268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:52.296268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.296268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:52.296268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:52.296268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.296268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:52.296268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:52.296268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.296268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:52.296268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:52.296268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:52.296582 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:52.296871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:52.296871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:52.296871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:56.936085 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:56.936636 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:56.936762 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:56.938306 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:56.938575 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:56.938770 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:56.938902 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:56.939053 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:56.939188 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:56.939327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:56.939487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:56.939640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:56.940345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:56.940345 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:56.940345 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:56.940345 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:56.940345 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:56.940345 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:56.940345 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:56.940345 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:56.941193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:56.941193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:56.941193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:56.941193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:56.941193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:56.941193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:56.941193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:56.941193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:56.941193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:56.941193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:56.941193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:56.941193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:56.941573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:56.941573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:56.941573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:56.941573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:56.941573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:56.941573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:56.941573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:56.941573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:56.941573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:56.941573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:56.941573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:56.941573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:56.941573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:56.941914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:56.941914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:56.942254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:56.942254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:56.942254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:56.942254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:56.942254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:56.942254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:56.942254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:56.942254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:56.942254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:56.942254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:56.942254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:56.942254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:56.942254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:56.942254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:56.942254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:56.942254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:56.942686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:56.943054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:56.943054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:56.943054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:57.003684 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:57.003934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:57.004018 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:57.004133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:45:57.004734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:57.004989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.005585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:57.005585 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:57.005585 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:57.005585 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:57.005585 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:57.005585 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:57.005585 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.005585 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:57.006169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.006169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:57.006169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:57.006169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:57.006169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.006169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:57.006169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:57.006169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:57.006169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.006169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:57.006169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:57.006169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:57.006169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:57.006169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:57.006169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:57.006566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:57.006566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:57.006566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:57.006566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:57.006566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:57.006566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:57.006566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:57.006566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.006566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:57.006828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.006828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:57.006828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:57.006828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:57.006828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.006828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:57.006828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:57.006828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:45:57.006828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:57.006828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:57.006828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.006828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:57.007082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:57.007082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:57.007082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.007082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:57.007082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:57.007082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:57.007082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:57.007082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:57.007082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:57.007082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:57.007082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:57.007082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:57.007082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:57.007082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:57.007291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:57.007291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:57.007291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.007291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:57.007628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.007628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:57.007628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:57.007628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:57.007628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:57.007628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:57.007628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.007628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:57.007628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:57.007628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.007628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:57.007628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:57.007628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.007628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:57.007628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:57.007628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:57.007969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:57.008400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:57.008400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:57.008400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:57.008400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:57.008400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:57.008400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.008400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:57.022282 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:57.022550 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:57.022675 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:57.022803 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:57.023280 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:57.023518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.024047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:57.024047 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:57.024047 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:57.024047 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:57.024047 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:57.024047 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:57.024047 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.024047 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:57.024965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.024965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:45:57.024965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:57.024965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:57.024965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.024965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:57.024965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:57.024965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:45:57.024965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.024965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:45:57.024965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:57.024965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:45:57.024965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:57.024965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:45:57.024965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:45:57.025283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:57.025283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:45:57.025283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:57.025283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:45:57.025283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:45:57.025283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:57.025283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:45:57.025283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.025283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:45:57.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:57.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:57.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:57.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:57.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:57.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:57.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:57.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:57.026151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:57.026151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:57.026151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.026151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:57.026151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:57.026151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:57.026151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:57.026151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:57.026151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:57.026151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:57.026151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:57.026151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:57.026151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:57.026151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:57.026433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:57.026433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:57.026433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.026433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.027089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:57.027439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:57.027802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:57.027802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:57.027802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.027802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:57.258977 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:57.259476 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:57.259653 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:57.259779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:57.260506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:57.260762 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:57.261472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.261909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:57.261909 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:57.261909 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:57.261909 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:57.261909 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:57.261909 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:57.261909 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.261909 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:57.263018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.263018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:57.263018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:57.263018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:57.263018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.263018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:57.263018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:57.263018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.263018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:57.263018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:57.263018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:57.263018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.264211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:57.264211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:57.264211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:57.264211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:57.264211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:57.264211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:57.264211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:57.264211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:57.264211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:57.264211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:57.264211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:57.264211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:57.264211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:57.264499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.264499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:57.264499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.264499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:57.264499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:57.264499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:57.264499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:57.264499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:57.264499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.264499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:57.264499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:57.264499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.264499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:57.264499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:57.264499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:57.264807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:57.265150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:57.265150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:57.265150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:57.265150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:57.265150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.265150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:45:57.265150 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:45:57.265150 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:45:57.265150 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:45:57.265150 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:45:57.310418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:45:57.310934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.311468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:45:57.311468 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:57.311468 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:45:57.311468 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:45:57.311468 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:57.311468 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:45:57.311468 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.311468 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:45:57.312271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.047 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.312271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:45:57.312271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:57.312271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:45:57.312271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.312271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:57.312271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:57.312271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.312271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:57.312271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:57.312271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:45:57.312271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.312779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:45:57.312779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:57.312779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:45:57.312779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:57.312779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:45:57.312779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:45:57.312779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:57.312779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:45:57.312779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:57.312779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:45:57.312779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:45:57.312779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:57.312779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:45:57.313282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.313282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:45:57.313454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.313454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:45:57.313454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:45:57.313454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:45:57.313454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:45:57.313454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:45:57.313454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.313454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:45:57.313454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:45:57.313454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.313454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:45:57.313454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:45:57.313454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.313454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:45:57.313454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:45:57.313454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:45:57.313764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:45:57.314120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:45:57.314120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:45:57.314120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:01.936605 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:01.937749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:01.937844 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:01.939671 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:01.939818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:01.939964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:01.940058 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:01.940194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:01.940285 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:01.940387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:01.940520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:01.940603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:01.941187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:01.941187 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:01.941187 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:01.941187 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:01.941187 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:01.941187 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:01.941187 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:01.941187 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:01.942029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:01.942029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:01.942029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:01.942029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:01.942029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:01.942029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:01.942029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:01.942029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:01.942029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:01.942029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:01.942029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:01.942029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:01.942309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:01.942309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:01.942309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:01.942309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:01.942309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:01.942309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:01.942309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:01.942309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:01.942309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:01.942309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:01.942309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:01.942309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:01.942309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:01.942637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:01.942637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:01.942948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:01.942948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:01.942948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:01.942948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:01.942948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:01.942948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:01.942948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:01.942948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:01.942948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:01.942948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:01.942948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:01.942948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:01.942948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:01.942948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:01.942948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:01.942948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:01.943600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:01.944003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:01.944003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:01.944003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:02.004148 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:02.004785 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:02.004785 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:02.004785 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:46:02.005112 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:02.005265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.006985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:02.007387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:02.007387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:02.007387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:02.007387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:02.007387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:02.007387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:02.007387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:02.007387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:02.007387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:02.007387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:02.007387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:02.007387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:02.007387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.007387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:02.007387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.007810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:02.007810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:02.007810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:02.007810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.007810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:02.007810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:02.007810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:46:02.007810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:02.007810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:02.007810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.007810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:02.007810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:02.007810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:02.008035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.008035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:02.008035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:02.008035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:02.008035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:02.008035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:02.008035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:02.008035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:02.008035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:02.008035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:02.008035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:02.008035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:02.008035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:02.008035 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:02.008324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.008324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:02.008324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.008324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:02.008324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:02.008324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:02.008324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:02.008324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:02.008324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.008324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:02.008324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:02.008324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.008324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:02.008324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:02.008324 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:02.008610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:02.008941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:02.008941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:02.008941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:02.008941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:02.008941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:02.008941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:02.008941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:02.008941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.008941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:02.022773 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:02.026803 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:02.026971 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:02.027137 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:02.027608 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:02.027879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.028472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:02.028472 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:02.028472 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:02.028472 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:02.028472 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:02.028472 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:02.028472 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.028472 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:02.029328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.029328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:02.029328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:02.029328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:02.029328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.029328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:02.029328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:02.029328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:02.029328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.029328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:02.029328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:02.029328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:02.029328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:02.029328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:02.029328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:02.029636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:02.029636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:02.029636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:02.029636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:02.029636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:02.029636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:02.029636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:02.029636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.029636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:02.030339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.030339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:02.030339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:02.030339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:02.030339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.030339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:02.030339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:02.030339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:02.030339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:02.030339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.030339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:02.030674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:02.030674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:02.030674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.030674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:02.030674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:02.030674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:02.030674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:02.030674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:02.030674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:02.030674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:02.030674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:02.030674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:02.030674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:02.030674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:02.031013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:02.031013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:02.031013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.031013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.031136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:02.031658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:02.032140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:02.032140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:02.032140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.032140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:02.259501 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:02.259948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:02.260095 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:02.260296 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:02.266837 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:02.267108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:02.267407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.268152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:02.268152 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:02.268152 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:02.268152 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:02.268152 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:02.268152 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:02.268152 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.268152 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:02.269334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.008 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.269334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:02.269334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:02.269334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:02.269334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.269334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:02.269334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:02.269334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.269334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:02.269334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:02.269334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:02.269334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.269608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:02.269608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:02.269608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:02.269608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:02.269608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:02.269608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:02.269608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:02.269608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:02.269608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:02.269608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:02.269608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:02.269608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:02.269608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:02.269822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.269822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:02.270536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.270536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:02.270536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:02.270536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:02.270536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:02.270536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:02.270536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.270536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:02.270536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:02.270536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.270536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:02.270536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:02.270536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.270536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:02.270536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:02.270536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:02.271157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:02.271157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.271157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:02.271733 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:02.272054 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:02.272190 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:02.273082 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:02.330054 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:02.330294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.330915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:02.330915 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:02.330915 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:02.330915 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:02.330915 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:02.330915 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:02.330915 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.330915 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:02.331839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.059 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.331839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:02.331839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:02.331839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:02.331839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.331839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:02.331839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:02.331839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.331839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:02.331839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:02.331839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:02.331839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.332060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:02.332060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:02.332060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:02.332060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:02.332060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:02.332060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:02.332060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:02.332060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:02.332060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:02.332060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:02.332060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:02.332060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:02.332060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:02.332326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.332326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:02.332932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.332932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:02.332932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:02.332932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:02.332932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:02.332932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:02.332932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.332932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:02.332932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:02.332932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.332932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:02.332932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:02.332932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.332932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:02.332932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:02.332932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:02.333231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:02.333624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:02.333624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:02.333624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:06.937890 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:06.939299 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:06.939299 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:06.939622 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:06.939830 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:06.940005 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:06.940174 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:06.940363 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:06.940558 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:06.940741 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:06.940908 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:06.941070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:06.941512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:06.941512 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:06.941512 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:06.941512 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:06.941512 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:06.941512 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:06.941512 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:06.941512 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:06.942290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:06.942290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:06.942290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:06.942290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:06.942290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:06.942290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:06.942290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:06.942290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:06.942290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:06.942290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:06.942290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:06.942290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:06.942624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:06.942624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:06.942624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:06.942624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:06.942624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:06.942624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:06.942624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:06.942624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:06.942624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:06.942624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:06.942624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:06.942624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:06.942624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:06.942853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:06.942853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:06.943703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:06.943703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:06.943703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:06.943703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:06.943703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:06.943703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:06.943703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:06.943703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:06.943703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:06.943703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:06.943703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:06.943703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:06.943703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:06.943703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:06.943703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:06.943703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:06.944013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:06.944303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:06.944303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:06.944303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:07.004580 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:07.004923 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:07.005042 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:07.005142 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:46:07.005614 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:07.005781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.006219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:07.006219 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:07.006219 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:07.006219 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:07.006219 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:07.006219 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:07.006219 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.006219 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:07.006867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.006867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:07.006867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:07.006867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:07.006867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.006867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:07.006867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:07.006867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:07.006867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.006867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:07.006867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:07.006867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:07.006867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:07.006867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:07.006867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:07.007382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:07.007382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:07.007382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:07.007382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:07.007382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:07.007382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:07.007382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:07.007382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.007382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:07.007706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.007706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:07.007706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:07.007706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:07.007706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.007706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:07.007706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:07.007706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:46:07.007706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:07.007706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:07.007706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.007706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:07.008341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:07.008341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:07.008341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.008341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:07.008341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:07.008341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:07.008341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:07.008341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:07.008341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:07.008341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:07.008341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:07.008341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:07.008341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:07.008341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:07.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:07.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:07.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:07.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:07.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:07.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:07.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:07.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:07.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:07.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:07.008694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:07.008910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:07.009199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:07.009199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:07.009199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:07.009199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:07.009199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:07.009199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:07.009199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:07.009199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:07.009199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:07.009199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:07.009199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.009199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:07.023233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:07.023484 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:07.023596 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:07.023716 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:07.024362 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:07.024600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.025024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:07.025024 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:07.025024 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:07.025024 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:07.025024 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:07.025024 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:07.025024 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.025024 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:07.025665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.025665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:07.025665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:07.025665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:07.025665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.025665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:07.025665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:07.025665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:07.025665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.025665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:07.025665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:07.025665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:07.025665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:07.025665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:07.025665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:07.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:07.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:07.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:07.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:07.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:07.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:07.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:07.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.025922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:07.026306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.026306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:07.026306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:07.026306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:07.026306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.026306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:07.026306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:07.026306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:07.026306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:07.026306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.026306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:07.026720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:07.026720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:07.026720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.026720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:07.026720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:07.026720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:07.026720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:07.026720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:07.026720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:07.026720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:07.026720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:07.026720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:07.026720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:07.026720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:07.027143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:07.027143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:07.027143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.027143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:07.027143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.027143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:07.027143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:07.027143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:07.027143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:07.027143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:07.027143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.027143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:07.027143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:07.027143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:07.027355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:07.027680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:07.027680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:07.027680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:07.027680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:07.027680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:07.027680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:07.027680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:07.027680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:07.027680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:07.027680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.027680 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:07.260069 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:07.260320 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:07.260430 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:07.260604 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:07.261098 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:07.261381 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:07.261656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.262408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:07.262408 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:07.262408 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:07.262408 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:07.262408 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:07.262408 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:07.262408 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.262408 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:07.263277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.263277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:07.263277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:07.263277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:07.263277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.263277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:07.263277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:07.263277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.263277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:07.263277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:07.263277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:07.263277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.263536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:07.263536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:07.263536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:07.263536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:07.263536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:07.263536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:07.263536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:07.263536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:07.263536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:07.263536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:07.263536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:07.263536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:07.263536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:07.263796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.263796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:07.263988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.263988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:07.263988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:07.263988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:07.263988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:07.263988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:07.263988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.263988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:07.263988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:07.263988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.263988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:07.263988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:07.263988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.263988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:07.263988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:07.263988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:07.264261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:07.264602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:07.264602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.264602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:07.272153 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:07.272510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:07.272637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:07.273215 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:07.316379 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:07.316757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.317406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:07.317406 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:07.317406 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:07.317406 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:07.317406 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:07.317406 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:07.317406 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.317406 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:07.318353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.045 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.318353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:07.318353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:07.318353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:07.318353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.318353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:07.318353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:07.318353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.318353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:07.318353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:07.318353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:07.318353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.318827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:07.318827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:07.318827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:07.318827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:07.318827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:07.318827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:07.318827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:07.318827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:07.318827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:07.318827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:07.318827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:07.318827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:07.318827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:07.319122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.319122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:07.319249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.319249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:07.319249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:07.319249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:07.319249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:07.319249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:07.319249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.319249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:07.319249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:07.319249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.319249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:07.319249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:07.319249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.319249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:07.319249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:07.319249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:07.320041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:07.320509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:07.320509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:07.320509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:11.938787 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:11.939459 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:11.939459 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:11.940206 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:11.940384 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:11.940566 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:11.940905 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:11.940905 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:11.941317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:11.941317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:11.941317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:11.941317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:11.944434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:11.944434 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:11.944434 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:11.944434 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:11.944434 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:11.944434 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:11.944434 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:11.944434 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:11.946912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.005 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:11.946912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:11.946912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:11.946912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:11.946912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:11.946912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:11.946912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:11.946912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:11.946912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:11.946912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:11.946912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:11.946912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:11.947706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:11.947706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:11.947706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:11.947706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:11.947706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:11.947706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:11.947706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:11.947706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:11.947706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:11.947706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:11.947706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:11.947706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:11.947706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:11.948391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:11.948391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:11.948391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:11.948391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:11.948391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:11.948391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:11.948391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:11.948391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:11.948391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:11.948391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:11.948391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:11.948391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:11.948391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:11.948391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:11.948391 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:11.949236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:11.950251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:11.950251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:11.950251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:11.950251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:11.950251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:11.950251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:12.005132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:12.005355 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:12.005490 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:12.005626 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:46:12.006557 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:12.006557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.006700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:12.006700 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:12.006700 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:12.006700 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:12.006700 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:12.006700 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:12.006700 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.006700 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:12.007258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.007258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:12.007258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:12.007258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:12.007258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.007258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:12.007258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:12.007258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:12.007258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.007258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:12.007258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:12.007258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:12.007258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:12.007258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:12.007258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:12.008988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:12.008988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:12.008988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:12.008988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:12.008988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:12.008988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:12.008988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:12.008988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.008988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:12.008988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.008988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:12.008988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:12.008988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:12.008988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.009785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:12.009785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:12.009785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:46:12.009785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:12.009785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:12.009785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.009785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:12.009785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:12.009785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:12.009785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.009785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:12.009785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:12.009785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:12.010724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:12.010724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:12.010724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:12.010724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:12.010724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:12.010724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:12.010724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:12.010724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:12.010724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:12.010724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:12.010724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.010724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:12.012090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.012090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:12.012090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:12.012090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:12.012090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:12.012090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:12.012090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.012090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:12.012090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:12.012090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.012090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:12.012090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:12.012090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.012090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:12.012090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:12.012090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:12.013235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:12.014415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:12.014415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:12.014415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:12.014415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:12.014415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:12.014415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.014415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:12.023723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:12.025950 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:12.026326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:12.026326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:12.026758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:12.026997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.027508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:12.027508 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:12.027508 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:12.027508 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:12.027508 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:12.027508 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:12.027508 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.027508 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:12.028396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.028396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:12.028396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:12.028396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:12.028396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.028396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:12.028396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:12.028396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:12.028396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.028396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:12.028396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:12.028396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:12.028396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:12.028396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:12.028396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:12.029799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:12.029799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:12.029799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:12.029799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:12.029799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:12.029799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:12.029799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:12.029799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.029799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:12.029799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.029799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:12.029799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:12.029799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:12.029799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.030786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:12.030786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:12.030786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:12.030786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:12.030786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.030786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:12.030786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:12.030786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:12.030786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.030786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:12.030786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:12.030786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:12.030786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:12.031185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:12.031185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:12.031185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:12.031185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:12.031185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:12.031185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:12.031185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:12.031185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:12.031185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:12.031185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.031185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:12.031185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:12.031496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:12.260518 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:12.260865 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:12.260966 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:12.261119 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:12.261403 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:12.261634 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:12.261829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.262224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:12.262224 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:12.262224 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:12.262224 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:12.262224 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:12.262224 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:12.262224 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.262224 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:12.262834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.262834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:12.262834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:12.262834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:12.262834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.262834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:12.262834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:12.262834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.262834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:12.262834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:12.262834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:12.262834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:12.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:12.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:12.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:12.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:12.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:12.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:12.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:12.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:12.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:12.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:12.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:12.263466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:12.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:12.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:12.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:12.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:12.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:12.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:12.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:12.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:12.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:12.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:12.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:12.264796 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:12.265271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:12.265271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:12.265271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:12.265271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:12.265271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.265271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:12.272596 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:12.272861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:12.272940 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:12.273511 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:12.301028 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:12.301193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.301621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:12.301621 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:12.301621 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:12.301621 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:12.301621 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:12.301621 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:12.301621 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.301621 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:12.302217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.302217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:12.302217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:12.302217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:12.302217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.302217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:12.302217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:12.302217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.302217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:12.302217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:12.302217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:12.302217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.302550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:12.302550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:12.302550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:12.302550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:12.302550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:12.302550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:12.302550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:12.302550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:12.302550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:12.302550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:12.302550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:12.302550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:12.302550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:12.302876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.302876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:12.302876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.302876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:12.302876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:12.302876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:12.302876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:12.302876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:12.302876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.302876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:12.302876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:12.302876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.302876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:12.302876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:12.302876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:12.303236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:12.303720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:12.303720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:12.303720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:12.303720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:12.303720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:12.303720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:16.939271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:16.939805 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:16.939805 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:16.940779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:16.940976 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:16.941134 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:16.941249 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:16.941599 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:16.941599 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:16.941599 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1cf3c33c-1249-4852-9c05-07626a6c0b31, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:16.941724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=7): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:16.941934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:16.942358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:16.942358 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:16.942358 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:16.942358 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:16.942358 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:16.942358 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:16.942358 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:16.942358 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:16.943784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:16.943784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:16.943784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:16.943784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:16.943784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:16.943784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:16.943784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:16.943784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:16.943784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:16.943784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:16.943784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:16.943784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:16.944021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:16.944021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:16.944021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:16.944021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:16.944021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:16.944021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:16.944021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:16.944021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:16.944021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:16.944021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:16.944021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:16.944021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:16.944021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:16.944385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:16.944385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:16.944891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:16.944891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:16.944891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:16.944891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:16.944891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:16.944891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:16.944891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:16.944891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:16.944891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:16.944891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:16.944891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:16.944891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:16.944891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:16.944891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:16.944891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:16.944891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:16.945521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:16.945897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:16.945897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:16.945897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:17.005394 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:17.005664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:17.005779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:17.005908 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:46:17.006358 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:17.006555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.007027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:17.007027 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:17.007027 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:17.007027 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:17.007027 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:17.007027 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:17.007027 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.007027 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:17.008198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.008198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:17.008198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:17.008198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:17.008198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.008198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:17.008198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:17.008198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:17.008198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.008198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:17.008198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:17.008198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:17.008198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:17.008198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:17.008198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:17.008637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:17.008637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:17.008637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:17.008637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:17.008637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:17.008637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:17.008637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:17.008637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.008637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:17.009034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.009034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:17.009034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:17.009034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:17.009034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.009034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:17.009034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:17.009034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:46:17.009034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:17.009034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:17.009034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.009034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:17.009271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:17.009271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:17.009271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.009271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:17.009271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:17.009271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:17.009271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:17.009271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:17.009271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:17.009271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:17.009271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:17.009271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:17.009271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:17.009271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:17.009535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:17.009535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:17.009535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.009535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:17.010084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.010084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:17.010084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:17.010084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:17.010084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:17.010084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:17.010084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.010084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:17.010084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:17.010084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.010084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:17.010084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:17.010084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.010084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:17.010084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:17.010084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:17.010341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:17.010653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:17.010653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:17.010653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:17.010653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:17.010653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:17.010653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.010653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:17.025884 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:17.026161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:17.026269 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:17.027285 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:17.029136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:17.029340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.029859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:17.029859 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:17.029859 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:17.029859 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:17.029859 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:17.029859 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:17.029859 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.029859 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:17.030719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.030719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:17.030719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:17.030719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:17.030719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.030719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:17.030719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:17.030719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:17.030719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.030719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:17.030719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:17.030719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:17.030719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:17.030719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:17.030719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:17.031045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:17.031045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:17.031045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:17.031045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:17.031045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:17.031045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:17.031045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:17.031045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.031045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:17.031635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.031635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:17.031635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:17.031635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:17.031635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.031635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:17.031635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:17.031635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:17.031635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:17.031635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.031635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:17.031933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:17.031933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:17.031933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.031933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:17.031933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:17.031933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:17.031933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:17.031933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:17.031933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:17.031933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:17.031933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:17.031933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:17.031933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:17.031933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:17.032173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:17.032173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:17.032173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.032173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.032716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:17.033045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:17.033365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:17.033365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:17.033365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.033365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:17.260920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:17.261551 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:17.261606 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:17.261791 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:17.263396 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:17.263677 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:17.263939 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.264483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:17.264483 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:17.264483 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:17.264483 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:17.264483 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:17.264483 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:17.264483 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.264483 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:17.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:17.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:17.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:17.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:17.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:17.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:17.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:17.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:17.265339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.265688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:17.265688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:17.265688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:17.265688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:17.265688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:17.265688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:17.265688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:17.265688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:17.265688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:17.265688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:17.265688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:17.265688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:17.265688 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:17.265926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.265926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:17.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:17.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:17.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:17.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:17.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:17.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:17.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:17.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:17.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:17.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:17.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:17.266367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:17.266755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:17.267112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:17.267112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.267112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:17.272932 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:17.273279 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:17.273380 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:17.274129 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:17.312213 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:17.312417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.312992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:17.312992 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:17.312992 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:17.312992 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:17.312992 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:17.312992 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:17.312992 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.312992 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:17.314024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.314024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:17.314024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:17.314024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:17.314024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.314024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:17.314024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:17.314024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.314024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:17.314024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:17.314024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:17.314024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.314306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:17.314306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:17.314306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:17.314306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:17.314306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:17.314306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:17.314306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:17.314306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:17.314306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:17.314306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:17.314306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:17.314306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:17.314306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:17.314570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.314570 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:17.315075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.315075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:17.315075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:17.315075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:17.315075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:17.315075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:17.315075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.315075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:17.315075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:17.315075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.315075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:17.315075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:17.315075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.315075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:17.315075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:17.315075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:17.315413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:17.315779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:17.315779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:17.315779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:21.939722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:21.940304 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:21.940304 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:21.941620 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:21.941805 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:21.942606 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-b3262570-4b85-46fc-af30-107155d4c53a, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:21.942773 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:21.943000 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:21.943162 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:21.943316 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=6): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:21.943507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:21.943986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:21.943986 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:21.943986 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:21.943986 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:21.943986 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:21.943986 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:21.943986 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:21.943986 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:21.944962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:21.944962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:21.944962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:21.944962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:21.944962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:21.944962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:21.944962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:21.944962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:21.944962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:21.944962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:21.944962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:21.944962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:21.945332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:21.945332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:21.945332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:21.945332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:21.945332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:21.945332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:21.945332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:21.945332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:21.945332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:21.945332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:21.945332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:21.945332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:21.945332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:21.945697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:21.945697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:21.946100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:21.946100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:21.946100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:21.946100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:21.946100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:21.946100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:21.946100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:21.946100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:21.946100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:21.946100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:21.946100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:21.946100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:21.946100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:21.946100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:21.946100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:21.946100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:21.946686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:21.947184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:21.947184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:21.947184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:22.005865 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:22.006186 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:22.006332 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:22.006459 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:46:22.006942 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:22.007174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.007690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:22.007690 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:22.007690 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:22.007690 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:22.007690 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:22.007690 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:22.007690 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.007690 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:22.008366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.008366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:22.008366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:22.008366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:22.008366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.008366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:22.008366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:22.008366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:22.008366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.008366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:22.008366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:22.008366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:22.008366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:22.008366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:22.008366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:22.008718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:22.008718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:22.008718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:22.008718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:22.008718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:22.008718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:22.008718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:22.008718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.008718 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:22.008989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.008989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:22.008989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:22.008989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:22.008989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.008989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:22.008989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:22.008989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:46:22.008989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:22.008989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:22.008989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.008989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:22.009231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:22.009231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:22.009231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.009231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:22.009231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:22.009231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:22.009231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:22.009231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:22.009231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:22.009231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:22.009231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:22.009231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:22.009231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:22.009231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:22.009511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:22.009511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:22.009511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.009511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:22.009686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.009686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:22.009686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:22.009686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:22.009686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:22.009686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:22.009686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.009686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:22.009686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:22.009686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.009686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:22.009686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:22.009686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.009686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:22.009686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:22.009686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:22.010154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:22.010486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:22.010486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:22.010486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:22.010486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:22.010486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:22.010486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.010486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:22.026277 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:22.026628 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:22.026724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:22.026856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:22.027273 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:22.027514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:22.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:22.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:22.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:22.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:22.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:22.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:22.028591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.028591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:22.028591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:22.028591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:22.028591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.028591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:22.028591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:22.028591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:22.028591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.028591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:22.028591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:22.028591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:22.028591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:22.028591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:22.028591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:22.028859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:22.028859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:22.028859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:22.028859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:22.028859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:22.028859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:22.028859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:22.028859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.028859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:22.029151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.029151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:22.029151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:22.029151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:22.029151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.029151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:22.029151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:22.029151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:22.029151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:22.029151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.029151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:22.029378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:22.029378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:22.029378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.029378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:22.029378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:22.029378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:22.029378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:22.029378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:22.029378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:22.029378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:22.029378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:22.029378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:22.029378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:22.029378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:22.029705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:22.029705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:22.029705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.029705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.029887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:22.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:22.030615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:22.030615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:22.030615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.030615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:22.261644 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:22.261947 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:22.262075 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:22.262275 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:22.262723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:22.262969 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:22.263164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.263636 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:22.263636 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:22.263636 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:22.263636 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:22.263636 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:22.263636 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:22.263636 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.263636 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:22.264380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.264380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:22.264380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:22.264380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:22.264380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.264380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:22.264380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:22.264380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.264380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:22.264380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:22.264380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:22.264380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.264711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:22.264711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:22.264711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:22.264711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:22.264711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:22.264711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:22.264711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:22.264711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:22.264711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:22.264711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:22.264711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:22.264711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:22.264711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:22.264942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.264942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:22.265199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.265199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:22.265199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:22.265199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:22.265199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:22.265199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:22.265199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.265199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:22.265199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:22.265199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.265199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:22.265199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:22.265199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.265199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:22.265199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:22.265199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:22.265534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:22.265941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:22.265941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.265941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:22.274643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:22.274643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:22.274643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:22.274643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:22.275390 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=236 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:46:22.308613 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:22.308819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.309319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:22.309319 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:22.309319 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:22.309319 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:22.309319 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:22.309319 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:22.309319 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.309319 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:22.310023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.310023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:22.310023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:22.310023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:22.310023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.310023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:22.310023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:22.310023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.310023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:22.310023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:22.310023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:22.310023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.310307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:22.310307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:22.310307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:22.310307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:22.310307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:22.310307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:22.310307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:22.310307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:22.310307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:22.310307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:22.310307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:22.310307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:22.310307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:22.310555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.310555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:22.310738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.310738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:22.310738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:22.310738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:22.310738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:22.310738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:22.310738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.310738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:22.310738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:22.310738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.310738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:22.310738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:22.310738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.310738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:22.310738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:22.310738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:22.311098 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:22.311502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:22.311502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:22.311502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:26.940298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:26.940813 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:26.940813 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:26.941790 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:26.942078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:26.942246 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-4a46adb1-95fd-4d79-911f-3caf4b050b46, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:26.942389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:26.942575 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:26.942720 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:26.942857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:26.943467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:26.943467 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:26.943467 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:26.943467 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:26.943467 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:26.943467 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:26.943467 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:26.943467 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:26.944137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:26.944137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:26.944137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:26.944137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:26.944137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:26.944137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:26.944137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:26.944137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:26.944137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:26.944137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:26.944137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:26.944137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:26.944516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:26.944516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:26.944516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:26.944516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:26.944516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:26.944516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:26.944516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:26.944516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:26.944516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:26.944516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:26.944516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:26.944516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:26.944516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:26.944930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:26.944930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:26.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:26.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:26.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:26.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:26.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:26.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:26.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:26.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:26.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:26.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:26.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:26.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:26.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:26.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:26.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:26.945004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:26.945608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:26.946128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:26.946128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:26.946128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:27.006234 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:27.006602 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:27.006695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:27.006794 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:46:27.007170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:27.007290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.007727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:27.007727 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:27.007727 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:27.007727 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:27.007727 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:27.007727 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:27.007727 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.007727 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:27.008261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:27.008530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:27.008530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:27.008530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:27.008530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:27.008530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:27.008530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:27.008530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:27.008530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.008530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:27.008881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.008881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:27.008881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:27.008881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:27.008881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.008881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:27.008881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:27.008881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:46:27.008881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:27.008881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:27.008881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.008881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:27.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:27.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:27.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:27.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:27.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:27.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:27.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:27.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:27.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:27.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:27.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:27.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:27.009104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:27.009334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:27.009334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:27.009334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.009334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:27.009581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.009581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:27.009581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:27.009581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:27.009581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:27.009581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:27.009581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.009581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:27.009581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:27.009581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.009581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:27.009581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:27.009581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.009581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:27.009581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:27.009581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:27.009857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:27.010177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:27.010177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:27.010177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:27.010177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:27.010177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:27.010177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.010177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:27.026811 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:27.027257 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:27.027391 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:27.027424 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:27.027774 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:27.028010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.028421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:27.028421 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:27.028421 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:27.028421 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:27.028421 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:27.028421 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:27.028421 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.028421 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:27.029038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.029038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:27.029038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:27.029038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:27.029038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.029038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:27.029038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:27.029038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:27.029038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.029038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:27.029038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:27.029038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:27.029038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:27.029038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:27.029038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:27.029332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:27.029332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:27.029332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:27.029332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:27.029332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:27.029332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:27.029332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:27.029332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.029332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:27.029679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.029679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:27.029679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:27.029679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:27.029679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.029679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:27.029679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:27.029679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:27.029679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:27.029679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.029679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:27.029879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:27.029879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:27.029879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.029879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:27.029879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:27.029879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:27.029879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:27.029879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:27.029879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:27.029879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:27.029879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:27.029879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:27.029879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:27.029879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:27.030167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:27.030167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:27.030167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.030167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.030512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:27.030959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:27.031281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:27.031281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:27.031281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.031281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:27.262019 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:27.262384 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:27.262500 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:27.262664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:27.263097 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:27.263318 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:27.263573 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.264020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:27.264020 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:27.264020 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:27.264020 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:27.264020 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:27.264020 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:27.264020 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.264020 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:27.264719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.264719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:27.264719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:27.264719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:27.264719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.264719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:27.264719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:27.264719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.264719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:27.264719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:27.264719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:27.264719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:27.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:27.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:27.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:27.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:27.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:27.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:27.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:27.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:27.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:27.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:27.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:27.265093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:27.265490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.265490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:27.265490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.265490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:27.265490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:27.265490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:27.265490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:27.265490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:27.265490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.265490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:27.265490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:27.265490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.265490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:27.265490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:27.265490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:27.265935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:27.266471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:27.266471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:27.266471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:27.266471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:27.266471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.266471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:27.273787 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:27.274136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:27.274229 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:27.274905 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:27.307552 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:27.307684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.308079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:27.308079 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:27.308079 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:27.308079 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:27.308079 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:27.308079 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:27.308079 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.308079 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:27.308770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.308770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:27.308770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:27.308770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:27.308770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.308770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:27.308770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:27.308770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.308770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:27.308770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:27.308770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:27.308770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.309079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:27.309079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:27.309079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:27.309079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:27.309079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:27.309079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:27.309079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:27.309079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:27.309079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:27.309079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:27.309079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:27.309079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:27.309079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:27.309292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.309292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:27.309502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.309502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:27.309502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:27.309502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:27.309502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:27.309502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:27.309502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.309502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:27.309502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:27.309502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.309502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:27.309502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:27.309502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.309502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:27.309502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:27.309502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:27.309987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:27.310555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:27.310555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:27.310555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:31.943811 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:31.944245 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:31.944400 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:31.945192 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:31.945337 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:31.945462 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:31.945590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:31.945697 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:31.945800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:31.946290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:31.946290 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:31.946290 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:31.946290 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:31.946290 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:31.946290 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:31.946290 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:31.946290 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:31.947657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:31.947657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:31.947657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:31.947657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:31.947657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:31.947657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:31.947657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:31.947657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:31.947657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:31.947657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:31.947657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:31.947657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:31.947989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:31.947989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:31.947989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:31.947989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:31.947989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:31.947989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:31.947989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:31.947989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:31.947989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:31.947989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:31.947989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:31.947989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:31.947989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:31.948267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:31.948267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:31.949613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:31.949613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:31.949613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:31.949613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:31.949613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:31.949613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:31.949613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:31.949613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:31.949613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:31.949613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:31.949613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:31.949613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:31.949613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:31.949613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:31.949613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:31.949613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:31.950794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:31.951184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:31.951184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:31.951184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:32.006669 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:32.007058 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:32.007201 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:32.007330 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:46:32.007799 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:32.007857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.008336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:32.008336 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:32.008336 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:32.008336 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:32.008336 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:32.008336 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:32.008336 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.008336 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:32.009327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.009327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:32.009327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:32.009327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:32.009327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.009327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:32.009327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:32.009327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:32.009327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.009327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:32.009327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:32.009327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:32.009327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:32.009327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:32.009327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:32.010013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:32.010013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:32.010013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:32.010013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:32.010013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:32.010013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:32.010013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:32.010013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.010013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:32.010013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.010013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:32.010013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:32.010013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:32.010013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.010431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:32.010431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:32.010431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:46:32.010431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:32.010431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:32.010431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.010431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:32.010431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:32.010431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:32.010431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.010431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:32.010431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:32.010431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:32.010957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:32.010957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:32.010957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:32.010957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:32.010957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:32.010957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:32.010957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:32.010957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:32.010957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:32.010957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:32.010957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.010957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:32.011297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.011297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:32.011297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:32.011297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:32.011297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:32.011297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:32.011297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.011297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:32.011297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:32.011297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.011297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:32.011297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:32.011297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.011297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:32.011297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:32.011297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:32.011742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:32.012276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:32.012276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:32.012276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:32.012276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:32.012276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:32.012276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.012276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:32.027292 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:32.027674 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:32.027799 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:32.027936 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:32.028241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:32.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.028855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:32.028855 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:32.028855 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:32.028855 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:32.028855 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:32.028855 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:32.028855 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.028855 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:32.029593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.029593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:32.029593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:32.029593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:32.029593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.029593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:32.029593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:32.029593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:32.029593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.029593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:32.029593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:32.029593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:32.029593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:32.029593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:32.029593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:32.029940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:32.029940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:32.029940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:32.029940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:32.029940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:32.029940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:32.029940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:32.029940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.029940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:32.030464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.030464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:32.030464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:32.030464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:32.030464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.030464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:32.030464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:32.030464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:32.030464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:32.030464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.030464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:32.030775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:32.030775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:32.030775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.030775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:32.030775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:32.030775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:32.030775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:32.030775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:32.030775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:32.030775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:32.030775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:32.030775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:32.030775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:32.030775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:32.031022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:32.031022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:32.031022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.031022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.031517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:32.031894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:32.032270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:32.032270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:32.032270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.032270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:32.262448 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:32.262841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:32.262959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:32.263117 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:32.263406 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:32.263650 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:32.263810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.264262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:32.264262 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:32.264262 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:32.264262 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:32.264262 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:32.264262 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:32.264262 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.264262 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:32.265036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.265036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:32.265036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:32.265036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:32.265036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.265036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:32.265036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:32.265036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.265036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:32.265036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:32.265036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:32.265036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.265299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:32.265299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:32.265299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:32.265299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:32.265299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:32.265299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:32.265299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:32.265299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:32.265299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:32.265299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:32.265299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:32.265299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:32.265299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:32.265525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.265525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:32.266143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.266143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:32.266143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:32.266143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:32.266143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:32.266143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:32.266143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.266143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:32.266143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:32.266143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.266143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:32.266143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:32.266143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.266143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:32.266143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:32.266143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:32.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:32.266909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:32.266909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.266909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:32.274163 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:32.274552 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:32.274651 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:32.275281 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:32.301327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:32.301527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.301942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:32.301942 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:32.301942 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:32.301942 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:32.301942 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:32.301942 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:32.301942 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.301942 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:32.302596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.302596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:32.302596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:32.302596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:32.302596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.302596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:32.302596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:32.302596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.302596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:32.302596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:32.302596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:32.302596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.302829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:32.302829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:32.302829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:32.302829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:32.302829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:32.302829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:32.302829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:32.302829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:32.302829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:32.302829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:32.302829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:32.302829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:32.302829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:32.303050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.303050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:32.303280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.303280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:32.303280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:32.303280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:32.303280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:32.303280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:32.303280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.303280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:32.303280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:32.303280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.303280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:32.303280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:32.303280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.303280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:32.303280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:32.303280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:32.303599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:32.303968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:32.303968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:32.303968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:36.944477 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:36.944998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:36.944998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:36.945788 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:36.946004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:36.946135 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:36.946243 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:36.946347 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:36.946481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:36.946959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:36.946959 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:36.946959 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:36.946959 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:36.946959 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:36.946959 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:36.946959 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:36.946959 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:36.947880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:36.947880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:36.947880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:36.947880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:36.947880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:36.947880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:36.947880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:36.947880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:36.947880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:36.947880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:36.947880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:36.947880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:36.948276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:36.948276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:36.948276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:36.948276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:36.948276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:36.948276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:36.948276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:36.948276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:36.948276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:36.948276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:36.948276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:36.948276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:36.948276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:36.948676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:36.948676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:36.948989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:36.948989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:36.948989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:36.948989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:36.948989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:36.948989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:36.948989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:36.948989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:36.948989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:36.948989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:36.948989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:36.948989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:36.948989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:36.948989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:36.948989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:36.948989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:36.949449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:36.950005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:36.950005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:36.950005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:37.007938 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:37.008375 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:37.008560 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:37.008719 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:46:37.009243 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:37.009465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.009979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:37.009979 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:37.009979 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:37.009979 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:37.009979 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:37.009979 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:37.009979 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.009979 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:37.010943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.010943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:37.010943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:37.010943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:37.010943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.010943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:37.010943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:37.010943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:37.010943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.010943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:37.010943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:37.010943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:37.010943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:37.010943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:37.010943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:37.011459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:37.011459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:37.011459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:37.011459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:37.011459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:37.011459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:37.011459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:37.011459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.011459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:37.011890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.011890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:37.011890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:37.011890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:37.011890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.011890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:37.011890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:37.011890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:46:37.011890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:37.011890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:37.011890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.011890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:37.012270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:37.012270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:37.012270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.012270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:37.012270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:37.012270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:37.012270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:37.012270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:37.012270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:37.012270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:37.012270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:37.012270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:37.012270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:37.012270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:37.012669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:37.012669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:37.012669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.012669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:37.013055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.013055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:37.013055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:37.013055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:37.013055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:37.013055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:37.013055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.013055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:37.013055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:37.013055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.013055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:37.013055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:37.013055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.013055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:37.013055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:37.013055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:37.013581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:37.014117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:37.014117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:37.014117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:37.014117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:37.014117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:37.014117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.014117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:37.027805 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:37.028274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:37.028473 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:37.028647 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:37.029151 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:37.029465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.030101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:37.030101 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:37.030101 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:37.030101 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:37.030101 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:37.030101 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:37.030101 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.030101 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:37.031113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.031113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:37.031113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:37.031113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:37.031113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.031113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:37.031113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:37.031113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:37.031113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.031113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:37.031113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:37.031113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:37.031113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:37.031113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:37.031113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:37.031601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:37.031601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:37.031601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:37.031601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:37.031601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:37.031601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:37.031601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:37.031601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.031601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:37.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:37.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:37.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:37.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:37.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:37.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:37.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:37.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:37.032567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:37.032567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:37.032567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.032567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:37.032567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:37.032567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:37.032567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:37.032567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:37.032567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:37.032567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:37.032567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:37.032567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:37.032567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:37.032567 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:37.033025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:37.033025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:37.033025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.033025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.033357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:37.033912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:37.034526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:37.034526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:37.034526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.034526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:37.262889 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:37.263444 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:37.263554 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:37.263702 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:37.264976 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:37.264976 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:37.264976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.265188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:37.265188 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:37.265188 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:37.265188 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:37.265188 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:37.265188 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:37.265188 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.265188 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:37.265748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.265748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:37.265748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:37.265748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:37.265748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.265748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:37.265748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:37.265748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.265748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:37.265748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:37.265748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:37.265748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.266150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:37.266150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:37.266150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:37.266150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:37.266150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:37.266150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:37.266150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:37.266150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:37.266150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:37.266150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:37.266150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:37.266150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:37.266150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:37.266524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.266524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:37.266524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.266524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:37.266524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:37.266524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:37.266524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:37.266524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:37.266524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.266524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:37.266524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:37.266524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.266524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:37.266524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:37.266524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:37.266896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:37.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:37.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:37.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:37.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:37.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.267217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:37.274586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:37.274858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:37.274982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:37.275555 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:37.302106 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:37.302106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.302298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:37.302298 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:37.302298 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:37.302298 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:37.302298 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:37.302298 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:37.302298 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.302298 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:37.303013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.303013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:37.303013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:37.303013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:37.303013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.303013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:37.303013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:37.303013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.303013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:37.303013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:37.303013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:37.303013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.303239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:37.303239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:37.303239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:37.303239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:37.303239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:37.303239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:37.303239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:37.303239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:37.303239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:37.303239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:37.303239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:37.303239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:37.303239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:37.303487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.303487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:37.303702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.303702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:37.303702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:37.303702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:37.303702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:37.303702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:37.303702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.303702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:37.303702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:37.303702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.303702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:37.303702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:37.303702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.303702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:37.303702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:37.303702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:37.304011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:37.304346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:37.304346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:37.304346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:41.944903 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:41.945374 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:41.945374 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:41.946053 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:41.946260 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:41.946397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:41.946568 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:41.946718 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:41.946844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:41.947348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:41.947348 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:41.947348 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:41.947348 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:41.947348 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:41.947348 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:41.947348 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:41.947348 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:41.948357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:41.948357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:41.948357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:41.948357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:41.948357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:41.948357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:41.948357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:41.948357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:41.948357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:41.948357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:41.948357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:41.948357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:41.948673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:41.948673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:41.948673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:41.948673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:41.948673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:41.948673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:41.948673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:41.948673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:41.948673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:41.948673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:41.948673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:41.948673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:41.948673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:41.948902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:41.948902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:41.949407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:41.949407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:41.949407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:41.949407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:41.949407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:41.949407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:41.949407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:41.949407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:41.949407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:41.949407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:41.949407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:41.949407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:41.949407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:41.949407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:41.949407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:41.949407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:41.949795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:41.950176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:41.950176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:41.950176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:42.008351 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:42.008734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:42.008844 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:42.008947 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:46:42.009305 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:42.009530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.010003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:42.010003 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:42.010003 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:42.010003 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:42.010003 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:42.010003 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:42.010003 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.010003 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:42.010555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.010555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:42.010555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:42.010555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:42.010555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.010555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:42.010555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:42.010555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:42.010555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.010555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:42.010555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:42.010555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:42.010555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:42.010555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:42.010555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:42.010854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:42.010854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:42.010854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:42.010854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:42.010854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:42.010854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:42.010854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:42.010854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.010854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:42.011283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.011283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:42.011283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:42.011283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:42.011283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.011283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:42.011283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:42.011283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:46:42.011283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:42.011283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:42.011283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.011283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:42.011523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:42.011523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:42.011523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.011523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:42.011523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:42.011523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:42.011523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:42.011523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:42.011523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:42.011523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:42.011523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:42.011523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:42.011523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:42.011523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:42.011809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:42.011809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:42.011809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.011809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:42.012279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.012279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:42.012279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:42.012279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:42.012279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:42.012279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:42.012279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.012279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:42.012279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:42.012279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.012279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:42.012279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:42.012279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.012279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:42.012279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:42.012279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:42.012608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:42.012960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:42.012960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:42.012960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:42.012960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:42.012960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:42.012960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.012960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:42.028203 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:42.028601 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:42.028709 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:42.028850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:42.029191 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:42.029369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.029817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:42.029817 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:42.029817 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:42.029817 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:42.029817 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:42.029817 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:42.029817 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.029817 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:42.030386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.030386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:42.030386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:42.030386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:42.030386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.030386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:42.030386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:42.030386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:42.030386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.030386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:42.030386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:42.030386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:42.030386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:42.030386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:42.030386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:42.030734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:42.030734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:42.030734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:42.030734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:42.030734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:42.030734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:42.030734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:42.030734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.030734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:42.031226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.031226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:42.031226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:42.031226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:42.031226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.031226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:42.031226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:42.031226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:42.031226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:42.031226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.031226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:42.031468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:42.031468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:42.031468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.031468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:42.031468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:42.031468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:42.031468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:42.031468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:42.031468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:42.031468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:42.031468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:42.031468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:42.031468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:42.031468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:42.031730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:42.031730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:42.031730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.031730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.032187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:42.032508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:42.032863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:42.032863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:42.032863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.032863 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:42.263562 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:42.263815 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:42.263900 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:42.264065 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:42.264724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:42.264999 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:42.265222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.265776 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:42.265776 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:42.265776 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:42.265776 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:42.265776 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:42.265776 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:42.265776 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.265776 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:42.266470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.266470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:42.266470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:42.266470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:42.266470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.266470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:42.266470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:42.266470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.266470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:42.266470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:42.266470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:42.266470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.266728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:42.266728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:42.266728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:42.266728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:42.266728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:42.266728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:42.266728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:42.266728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:42.266728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:42.266728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:42.266728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:42.266728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:42.266728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:42.266936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.266936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:42.267403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.267403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:42.267403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:42.267403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:42.267403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:42.267403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:42.267403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.267403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:42.267403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:42.267403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.267403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:42.267403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:42.267403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.267403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:42.267403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:42.267403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:42.267821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:42.268284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:42.268284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.268284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:42.274933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:42.275151 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:42.275254 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:42.276046 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:42.308066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:42.308314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.308969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:42.308969 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:42.308969 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:42.308969 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:42.308969 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:42.308969 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:42.308969 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.308969 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:42.309876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.309876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:42.309876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:42.309876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:42.309876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.309876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:42.309876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:42.309876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.309876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:42.309876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:42.309876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:42.309876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.310156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:42.310156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:42.310156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:42.310156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:42.310156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:42.310156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:42.310156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:42.310156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:42.310156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:42.310156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:42.310156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:42.310156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:42.310156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:42.310471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.310471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:42.310887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.310887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:42.310887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:42.310887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:42.310887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:42.310887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:42.310887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.310887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:42.310887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:42.310887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.310887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:42.310887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:42.310887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.310887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:42.310887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:42.310887 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:42.311229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:42.311580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:42.311580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:42.311580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:46.945327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:46.945860 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:46.945860 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:46.946523 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:46.946717 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:46.946855 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:46.946973 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:46.947076 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:46.947177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:46.947634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:46.947634 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:46.947634 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:46.947634 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:46.947634 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:46.947634 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:46.947634 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:46.947634 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:46.948418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:46.948418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:46.948418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:46.948418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:46.948418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:46.948418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:46.948418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:46.948418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:46.948418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:46.948418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:46.948418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:46.948418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:46.948691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:46.948691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:46.948691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:46.948691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:46.948691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:46.948691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:46.948691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:46.948691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:46.948691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:46.948691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:46.948691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:46.948691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:46.948691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:46.948932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:46.948932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:46.949250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:46.949250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:46.949250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:46.949250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:46.949250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:46.949250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:46.949250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:46.949250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:46.949250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:46.949250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:46.949250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:46.949250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:46.949250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:46.949250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:46.949250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:46.949250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:46.949591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:46.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:46.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:46.949933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:47.008856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:47.009284 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:47.009387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:47.009511 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:46:47.009951 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:47.010128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.010615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:47.010615 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:47.010615 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:47.010615 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:47.010615 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:47.010615 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:47.010615 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.010615 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:47.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:47.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:47.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:47.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:47.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:47.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:47.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:47.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:47.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:47.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:47.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:47.011357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:47.011726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:47.011726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:47.011726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:47.011726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:47.011726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:47.011726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:47.011726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:47.011726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.011726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:47.012101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.012101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:47.012101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:47.012101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:47.012101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.012101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:47.012101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:47.012101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:46:47.012101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:47.012101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:47.012101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.012101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:47.012357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:47.012357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:47.012357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.012357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:47.012357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:47.012357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:47.012357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:47.012357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:47.012357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:47.012357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:47.012357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:47.012357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:47.012357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:47.012357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:47.012662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:47.012662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:47.012662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.012662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:47.012977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.012977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:47.012977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:47.012977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:47.012977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:47.012977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:47.012977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.012977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:47.012977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:47.012977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.012977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:47.012977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:47.012977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.012977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:47.012977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:47.012977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:47.013308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:47.013676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:47.013676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:47.013676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:47.013676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:47.013676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:47.013676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.013676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:47.028602 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:47.029016 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:47.029137 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:47.029262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:47.029769 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:47.029981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.030466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:47.030466 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:47.030466 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:47.030466 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:47.030466 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:47.030466 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:47.030466 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.030466 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:47.031238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.031238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:47.031238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:47.031238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:47.031238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.031238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:47.031238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:47.031238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:47.031238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.031238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:47.031238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:47.031238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:47.031238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:47.031238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:47.031238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:47.031502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:47.031502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:47.031502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:47.031502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:47.031502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:47.031502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:47.031502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:47.031502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.031502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:47.032027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.032027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:47.032027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:47.032027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:47.032027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.032027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:47.032027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:47.032027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:47.032027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:47.032027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.032027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:47.032225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:47.032225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:47.032225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.032225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:47.032225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:47.032225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:47.032225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:47.032225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:47.032225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:47.032225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:47.032225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:47.032225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:47.032225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:47.032225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:47.032519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:47.032519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:47.032519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.032519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.032940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:47.033222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:47.033521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:47.033521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:47.033521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.033521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:47.263939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:47.264192 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:47.264288 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:47.264420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:47.264893 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:47.265116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:47.265289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.265768 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:47.265768 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:47.265768 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:47.265768 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:47.265768 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:47.265768 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:47.265768 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.265768 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:47.266427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.266427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:47.266427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:47.266427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:47.266427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.266427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:47.266427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:47.266427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.266427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:47.266427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:47.266427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:47.266427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.266777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:47.266777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:47.266777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:47.266777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:47.266777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:47.266777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:47.266777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:47.266777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:47.266777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:47.266777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:47.266777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:47.266777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:47.266777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:47.267155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.267155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:47.267155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.267155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:47.267155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:47.267155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:47.267155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:47.267155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:47.267155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.267155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:47.267155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:47.267155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.267155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:47.267155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:47.267155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:47.267621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:47.267945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:47.267945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:47.267945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:47.267945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:47.267945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.267945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:47.275270 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:47.275595 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:47.275686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:47.276269 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:47.303786 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:47.304017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.304589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:47.304589 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:47.304589 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:47.304589 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:47.304589 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:47.304589 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:47.304589 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.304589 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:47.305252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.305252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:47.305252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:47.305252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:47.305252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.305252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:47.305252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:47.305252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.305252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:47.305252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:47.305252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:47.305252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.305523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:47.305523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:47.305523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:47.305523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:47.305523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:47.305523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:47.305523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:47.305523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:47.305523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:47.305523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:47.305523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:47.305523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:47.305523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:47.305765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.305765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:47.305960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.305960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:47.305960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:47.305960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:47.305960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:47.305960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:47.305960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.305960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:47.305960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:47.305960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.305960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:47.305960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:47.305960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.305960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:47.305960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:47.305960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:47.306296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:47.306668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:47.306668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:47.306668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:51.945684 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:51.946203 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:51.946203 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:51.947021 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:51.947200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:51.947315 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:51.947422 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:51.947559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:51.947675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:51.948168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:51.948168 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:51.948168 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:51.948168 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:51.948168 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:51.948168 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:51.948168 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:51.948168 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:51.949055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:51.949055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:51.949055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:51.949055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:51.949055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:51.949055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:51.949055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:51.949055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:51.949055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:51.949055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:51.949055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:51.949055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:51.949349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:51.949349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:51.949349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:51.949349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:51.949349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:51.949349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:51.949349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:51.949349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:51.949349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:51.949349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:51.949349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:51.949349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:51.949349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:51.949666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:51.949666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:51.949992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:51.949992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:51.949992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:51.949992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:51.949992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:51.949992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:51.949992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:51.949992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:51.949992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:51.949992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:51.949992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:51.949992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:51.949992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:51.949992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:51.949992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:51.949992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:51.950323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:51.950777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:51.950777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:51.950777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:52.009377 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:52.009730 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:52.009836 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:52.009935 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:46:52.010413 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:52.010630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.011116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:52.011116 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:52.011116 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:52.011116 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:52.011116 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:52.011116 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:52.011116 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.011116 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:52.011917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.011917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:52.011917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:52.011917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:52.011917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.011917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:52.011917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:52.011917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:52.011917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.011917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:52.011917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:52.011917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:52.011917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:52.011917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:52.011917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:52.012224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:52.012224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:52.012224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:52.012224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:52.012224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:52.012224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:52.012224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:52.012224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.012224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:52.012523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.012523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:52.012523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:52.012523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:52.012523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.012523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:52.012523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:52.012523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:46:52.012523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:52.012523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:52.012523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.012523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:52.012803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:52.012803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:52.012803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.012803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:52.012803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:52.012803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:52.012803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:52.012803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:52.012803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:52.012803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:52.012803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:52.012803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:52.012803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:52.012803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:52.013031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:52.013031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:52.013031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.013031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:52.013365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.013365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:52.013365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:52.013365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:52.013365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:52.013365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:52.013365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.013365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:52.013365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:52.013365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.013365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:52.013365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:52.013365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.013365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:52.013365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:52.013365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:52.013669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:52.013982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:52.013982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:52.013982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:52.013982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:52.013982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:52.013982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.013982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:52.029035 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:52.029322 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:52.029471 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:52.029637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:52.029992 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:52.030189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.030596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:52.030596 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:52.030596 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:52.030596 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:52.030596 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:52.030596 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:52.030596 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.030596 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:52.031338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.031338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:52.031338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:52.031338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:52.031338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.031338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:52.031338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:52.031338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:52.031338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.031338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:52.031338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:52.031338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:52.031338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:52.031338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:52.031338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:52.031800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:52.031800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:52.031800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:52.031800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:52.031800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:52.031800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:52.031800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:52.031800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.031800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:52.032358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.032358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:52.032358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:52.032358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:52.032358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.032358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:52.032358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:52.032358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:52.032358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:52.032358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.032358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:52.032807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:52.032807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:52.032807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.032807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:52.032807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:52.032807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:52.032807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:52.032807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:52.032807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:52.032807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:52.032807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:52.032807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:52.032807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:52.032807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:52.033260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:52.033260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:52.033260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.033260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.033504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:52.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:52.034209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:52.034209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:52.034209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.034209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:52.264340 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:52.264679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:52.264784 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:52.264977 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:52.265518 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:52.265775 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:52.265940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:52.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:52.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:52.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:52.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:52.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:52.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.266469 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:52.267198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.267198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:52.267198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:52.267198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:52.267198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.267198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:52.267198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:52.267198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.267198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:52.267198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:52.267198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:52.267198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.267520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:52.267520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:52.267520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:52.267520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:52.267520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:52.267520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:52.267520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:52.267520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:52.267520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:52.267520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:52.267520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:52.267520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:52.267520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:52.267859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.267859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:52.267963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.267963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:52.267963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:52.267963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:52.267963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:52.267963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:52.267963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.267963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:52.267963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:52.267963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.267963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:52.267963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:52.267963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.267963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:52.267963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:52.267963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:52.268337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:52.268811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:52.268811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.268811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:52.275594 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:52.275820 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:52.275934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:52.276747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:52.308554 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:52.308792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.309353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:52.309353 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:52.309353 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:52.309353 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:52.309353 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:52.309353 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:52.309353 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.309353 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:52.310294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.310294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:52.310294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:52.310294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:52.310294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.310294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:52.310294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:52.310294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.310294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:52.310294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:52.310294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:52.310294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.310778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:52.310778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:52.310778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:52.310778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:52.310778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:52.310778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:52.310778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:52.310778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:52.310778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:52.310778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:52.310778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:52.310778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:52.310778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:52.311145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.311145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:52.311235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.311235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:52.311235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:52.311235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:52.311235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:52.311235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:52.311235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.311235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:52.311235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:52.311235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.311235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:52.311235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:52.311235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.311235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:52.311235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:52.311235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:52.311717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:52.312250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:52.312250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:52.312250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:56.946058 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:56.946486 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:56.946688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:56.947550 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:56.947720 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:56.947845 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:56.947957 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:56.948065 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:56.948172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:56.950349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:56.950349 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:56.950349 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:56.950349 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:56.950349 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:56.950349 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:56.950349 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:56.950349 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:56.951623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:56.951623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:56.951623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:56.951623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:56.951623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:56.951623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:56.951623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:56.951623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:56.951623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:56.951623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:56.951623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:56.951623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:56.951998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:56.951998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:56.951998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:56.951998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:56.951998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:56.951998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:56.951998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:56.951998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:56.951998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:56.951998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:56.951998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:56.951998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:56.951998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:56.952326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:56.952326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:56.952550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:56.952550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:56.952550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:56.952550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:56.952550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:56.952550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:56.952550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:56.952550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:56.952550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:56.952550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:56.952550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:56.952550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:56.952550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:56.952550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:56.952550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:56.952550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:56.952971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:56.953555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:56.953555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:56.953555 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:57.009915 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:57.010347 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:57.010481 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:57.010641 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:46:57.010993 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:57.011157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.011743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:57.011743 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:57.011743 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:57.011743 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:57.011743 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:57.011743 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:57.011743 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.011743 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:57.013907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.013907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:57.013907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:57.013907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:57.013907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.013907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:57.013907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:57.013907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:57.013907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.013907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:57.013907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:57.013907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:57.013907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:57.013907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:57.013907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:57.014209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:57.014209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:57.014209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:57.014209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:57.014209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:57.014209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:57.014209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:57.014209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.014209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:57.014761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.014761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:57.014761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:57.014761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:57.014761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.014761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:57.014761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:57.014761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:46:57.014761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:57.014761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:57.014761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.014761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:57.015066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:57.015066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:57.015066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.015066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:57.015066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:57.015066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:57.015066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:57.015066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:57.015066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:57.015066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:57.015066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:57.015066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:57.015066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:57.015066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:57.015316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:57.015316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:57.015316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.015316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:57.015809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.015809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:57.015809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:57.015809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:57.015809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:57.015809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:57.015809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.015809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:57.015809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:57.015809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.015809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:57.015809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:57.015809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.015809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:57.015809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:57.015809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:57.016105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:57.016418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:57.016418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:57.016418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:57.016418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:57.016418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:57.016418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.016418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:57.029585 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:57.029941 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:57.030049 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:57.030193 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:57.030670 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:57.030861 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.031317 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:57.031317 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:57.031317 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:57.031317 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:57.031317 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:57.031317 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:57.031317 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.031317 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:57.032145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.032145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:46:57.032145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:57.032145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:57.032145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.032145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:57.032145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:57.032145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:46:57.032145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.032145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:46:57.032145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:57.032145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:46:57.032145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:57.032145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:46:57.032145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:46:57.032463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:57.032463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:46:57.032463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:57.032463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:46:57.032463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:46:57.032463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:57.032463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:46:57.032463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.032463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:46:57.033030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.033030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:57.033030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:57.033030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:57.033030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.033030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:57.033030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:57.033030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:57.033030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:57.033030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.033030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:57.033296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:57.033296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:57.033296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.033296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:57.033296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:57.033296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:57.033296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:57.033296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:57.033296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:57.033296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:57.033296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:57.033296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:57.033296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:57.033296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:57.033574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:57.033574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:57.033574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.033574 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.034107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:57.034523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:57.034894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:57.034894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:57.034894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.034894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:57.264844 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:57.265191 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:57.265285 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:57.265429 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:57.265768 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:57.265959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:57.266123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.266519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:57.266519 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:57.266519 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:57.266519 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:57.266519 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:57.266519 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:57.266519 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.266519 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:57.267140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.267140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:57.267140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:57.267140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:57.267140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.267140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:57.267140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:57.267140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.267140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:57.267140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:57.267140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:57.267140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.267368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:57.267368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:57.267368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:57.267368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:57.267368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:57.267368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:57.267368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:57.267368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:57.267368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:57.267368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:57.267368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:57.267368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:57.267368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:57.267651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.267651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:57.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:57.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:57.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:57.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:57.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:57.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:57.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:57.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:57.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:57.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:57.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:57.267882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:57.268179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:57.268533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:57.268533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.268533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:46:57.275967 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:46:57.276211 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:46:57.276299 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:46:57.276937 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:46:57.303731 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:46:57.303872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.304381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:46:57.304381 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:57.304381 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:46:57.304381 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:46:57.304381 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:57.304381 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:46:57.304381 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.304381 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:46:57.305039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.305039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:46:57.305039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:57.305039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:46:57.305039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.305039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:57.305039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:57.305039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.305039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:57.305039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:57.305039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:46:57.305039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.305290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:46:57.305290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:57.305290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:46:57.305290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:57.305290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:46:57.305290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:46:57.305290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:57.305290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:46:57.305290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:57.305290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:46:57.305290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:46:57.305290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:57.305290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:46:57.305553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.305553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:46:57.305747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.305747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:46:57.305747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:46:57.305747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:46:57.305747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:46:57.305747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:46:57.305747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.305747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:46:57.305747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:46:57.305747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.305747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:46:57.305747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:46:57.305747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.305747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:46:57.305747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:46:57.305747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:46:57.306107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:46:57.306407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:46:57.306407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:46:57.306407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:01.946505 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:01.949873 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:01.949975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:01.950826 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:01.950968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:01.951077 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:01.951185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:01.951287 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:01.951385 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:01.952787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:01.952787 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:01.952787 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:01.952787 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:01.952787 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:01.952787 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:01.952787 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:01.952787 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:01.953394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:01.953394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:01.953394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:01.953394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:01.953394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:01.953394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:01.953394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:01.953394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:01.953394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:01.953394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:01.953394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:01.953394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:01.953709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:01.953709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:01.953709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:01.953709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:01.953709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:01.953709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:01.953709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:01.953709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:01.953709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:01.953709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:01.953709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:01.953709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:01.953709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:01.953933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:01.953933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:01.954106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:01.954106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:01.954106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:01.954106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:01.954106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:01.954106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:01.954106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:01.954106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:01.954106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:01.954106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:01.954106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:01.954106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:01.954106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:01.954106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:01.954106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:01.954106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:01.954599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:01.954925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:01.954925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:01.954925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:02.010382 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:02.010841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:02.010951 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:02.011051 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:47:02.011418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:02.011720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.012634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:02.012634 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:02.012634 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:02.012634 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:02.012634 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:02.012634 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:02.012634 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.012634 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:02.013374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.013374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:02.013374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:02.013374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:02.013374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.013374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:02.013374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:02.013374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:02.013374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.013374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:02.013374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:02.013374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:02.013374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:02.013374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:02.013374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:02.013868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:02.013868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:02.013868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:02.013868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:02.013868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:02.013868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:02.013868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:02.013868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.013868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:02.013868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.013868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:02.013868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:02.013868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:02.013868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.014256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:02.014256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:02.014256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:47:02.014256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:02.014256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:02.014256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.014256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:02.014256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:02.014256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:02.014256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.014256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:02.014256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:02.014256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:02.014598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:02.014598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:02.014598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:02.014598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:02.014598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:02.014598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:02.014598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:02.014598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:02.014598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:02.014598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:02.014598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.014598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:02.015455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.015455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:02.015455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:02.015455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:02.015455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:02.015455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:02.015455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.015455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:02.015455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:02.015455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.015455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:02.015455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:02.015455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.015455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:02.015455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:02.015455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:02.017152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:02.017699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:02.017699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:02.017699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:02.017699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:02.017699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:02.017699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.017699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:02.030842 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:02.031259 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:02.031361 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:02.031476 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:02.032581 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:02.032581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.033750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:02.033750 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:02.033750 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:02.033750 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:02.033750 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:02.033750 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:02.033750 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.033750 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:02.034553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.034553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:02.034553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:02.034553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:02.034553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.034553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:02.034553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:02.034553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:02.034553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.034553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:02.034553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:02.034553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:02.034553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:02.034553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:02.034553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:02.034876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:02.034876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:02.034876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:02.034876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:02.034876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:02.034876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:02.034876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:02.034876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.034876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:02.035480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.035480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:02.035480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:02.035480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:02.035480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.035480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:02.035480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:02.035480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:02.035480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:02.035480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.035480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:02.035710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:02.035710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:02.035710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.035710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:02.035710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:02.035710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:02.035710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:02.035710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:02.035710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:02.035710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:02.035710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:02.035710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:02.035710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:02.035710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:02.035915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:02.035915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:02.035915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.035915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.036922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:02.037220 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:02.037522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:02.037522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:02.037522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.037522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:02.265220 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:02.266214 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:02.266214 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:02.266214 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:02.267047 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:02.267473 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:02.268236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.268600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:02.268600 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:02.268600 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:02.268600 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:02.268600 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:02.268600 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:02.268600 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.268600 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:02.270580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.270580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:02.270580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:02.270580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:02.270580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.270580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:02.270580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:02.270580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.270580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:02.270580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:02.270580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:02.270580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.271066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:02.271066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:02.271066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:02.271066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:02.271066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:02.271066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:02.271066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:02.271066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:02.271066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:02.271066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:02.271066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:02.271066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:02.271066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:02.271451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.271451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:02.276744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:02.277078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:02.277393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:02.277393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.277393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:02.277393 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:02.278616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:02.278616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:02.278854 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:02.317104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:02.317259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.317687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:02.317687 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:02.317687 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:02.317687 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:02.317687 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:02.317687 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:02.317687 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.317687 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:02.319177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.319177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:02.319177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:02.319177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:02.319177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.319177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:02.319177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:02.319177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.319177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:02.319177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:02.319177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:02.319177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.319584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:02.319584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:02.319584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:02.319584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:02.319584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:02.319584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:02.319584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:02.319584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:02.319584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:02.319584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:02.319584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:02.319584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:02.319584 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:02.319891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.319891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:02.319891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.319891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:02.319891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:02.319891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:02.319891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:02.319891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:02.319891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.319891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:02.319891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:02.319891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.319891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:02.319891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:02.319891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:02.320211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:02.320572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:02.320572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:02.320572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:02.320572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:02.320572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:02.320572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:06.949952 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:06.950572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:06.950572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:06.951093 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:06.951229 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:06.951341 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:06.951451 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:06.951571 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:06.951675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:06.952089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:06.952089 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:06.952089 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:06.952089 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:06.952089 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:06.952089 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:06.952089 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:06.952089 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:06.952819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:06.952819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:06.952819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:06.952819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:06.952819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:06.952819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:06.952819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:06.952819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:06.952819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:06.952819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:06.952819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:06.952819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:06.953073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:06.953073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:06.953073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:06.953073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:06.953073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:06.953073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:06.953073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:06.953073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:06.953073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:06.953073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:06.953073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:06.953073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:06.953073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:06.953364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:06.953364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:06.953800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:06.953800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:06.953800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:06.953800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:06.953800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:06.953800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:06.953800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:06.953800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:06.953800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:06.953800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:06.953800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:06.953800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:06.953800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:06.953800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:06.953800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:06.953800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:06.954089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:06.954467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:06.954467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:06.954467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:07.010769 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:07.011186 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:07.011260 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:07.011360 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:47:07.011739 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:07.011929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.012417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:07.012417 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:07.012417 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:07.012417 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:07.012417 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:07.012417 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:07.012417 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.012417 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:07.013012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.013012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:07.013012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:07.013012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:07.013012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.013012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:07.013012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:07.013012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:07.013012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.013012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:07.013012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:07.013012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:07.013012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:07.013012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:07.013012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:07.013492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:07.013492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:07.013492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:07.013492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:07.013492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:07.013492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:07.013492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:07.013492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.013492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:07.013744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.013744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:07.013744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:07.013744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:07.013744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.013744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:07.013744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:07.013744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:47:07.013744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:07.013744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:07.013744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.013744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:07.013978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:07.013978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:07.013978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.013978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:07.013978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:07.013978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:07.013978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:07.013978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:07.013978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:07.013978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:07.013978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:07.013978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:07.013978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:07.013978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:07.014228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:07.014228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:07.014228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.014228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:07.014532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.014532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:07.014532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:07.014532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:07.014532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:07.014532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:07.014532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.014532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:07.014532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:07.014532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.014532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:07.014532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:07.014532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.014532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:07.014532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:07.014532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:07.014991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:07.015557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:07.015557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:07.015557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:07.015557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:07.015557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:07.015557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.015557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:07.031364 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:07.031648 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:07.031769 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:07.031920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:07.032367 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:07.032464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.032903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:07.032903 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:07.032903 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:07.032903 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:07.032903 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:07.032903 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:07.032903 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.032903 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:07.033491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.033491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:07.033491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:07.033491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:07.033491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.033491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:07.033491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:07.033491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:07.033491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.033491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:07.033491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:07.033491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:07.033491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:07.033491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:07.033491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:07.033798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:07.033798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:07.033798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:07.033798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:07.033798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:07.033798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:07.033798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:07.033798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.033798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:07.034217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.034217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:07.034217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:07.034217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:07.034217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.034217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:07.034217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:07.034217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:07.034217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:07.034217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.034217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:07.034494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:07.034494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:07.034494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.034494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:07.034494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:07.034494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:07.034494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:07.034494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:07.034494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:07.034494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:07.034494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:07.034494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:07.034494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:07.034494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:07.034774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:07.034774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:07.034774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.034774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.035221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:07.035600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:07.036068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:07.036068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:07.036068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.036068 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:07.265770 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:07.266140 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:07.266288 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:07.266455 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:07.266827 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:07.267037 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:07.267210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.267703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:07.267703 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:07.267703 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:07.267703 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:07.267703 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:07.267703 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:07.267703 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.267703 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:07.268387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.268387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:07.268387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:07.268387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:07.268387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.268387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:07.268387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:07.268387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.268387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:07.268387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:07.268387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:07.268387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.268823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:07.268823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:07.268823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:07.268823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:07.268823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:07.268823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:07.268823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:07.268823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:07.268823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:07.268823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:07.268823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:07.268823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:07.268823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:07.269192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.269192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:07.269192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.269192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:07.269192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:07.269192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:07.269192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:07.269192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:07.269192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.269192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:07.269192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:07.269192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.269192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:07.269192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:07.269192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:07.269644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:07.270149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:07.270149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:07.270149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:07.270149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:07.270149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.270149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:07.277881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:07.278143 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:07.278241 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:07.278974 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:07.313016 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:07.313255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.313872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:07.313872 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:07.313872 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:07.313872 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:07.313872 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:07.313872 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:07.313872 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.313872 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:07.315009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.036 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.315009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:07.315009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:07.315009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:07.315009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.315009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:07.315009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:07.315009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.315009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:07.315009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:07.315009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:07.315009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.315407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:07.315407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:07.315407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:07.315407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:07.315407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:07.315407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:07.315407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:07.315407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:07.315407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:07.315407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:07.315407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:07.315407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:07.315407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:07.315779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.315779 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:07.316158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.316158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:07.316158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:07.316158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:07.316158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:07.316158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:07.316158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.316158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:07.316158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:07.316158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.316158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:07.316158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:07.316158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.316158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:07.316158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:07.316158 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:07.316661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:07.317179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:07.317179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:07.317179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:11.950340 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:11.950814 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:11.950913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:11.951657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:11.951852 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:11.951985 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:11.952096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:11.952200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:11.952307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:11.952827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:11.952827 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:11.952827 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:11.952827 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:11.952827 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:11.952827 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:11.952827 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:11.952827 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:11.953561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:11.953561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:11.953561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:11.953561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:11.953561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:11.953561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:11.953561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:11.953561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:11.953561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:11.953561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:11.953561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:11.953561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:11.953809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:11.953809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:11.953809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:11.953809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:11.953809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:11.953809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:11.953809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:11.953809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:11.953809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:11.953809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:11.953809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:11.953809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:11.953809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:11.954074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:11.954074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:11.954257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:11.954257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:11.954257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:11.954257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:11.954257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:11.954257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:11.954257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:11.954257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:11.954257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:11.954257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:11.954257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:11.954257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:11.954257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:11.954257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:11.954257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:11.954257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:11.954593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:11.954925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:11.954925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:11.954925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:12.011196 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:12.011621 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:12.011794 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:12.011966 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:47:12.013380 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:12.013662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.014403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:12.014403 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:12.014403 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:12.014403 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:12.014403 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:12.014403 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:12.014403 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.014403 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:12.016154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.016154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:12.016154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:12.016154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:12.016154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.016154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:12.016154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:12.016154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:12.016154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.016154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:12.016154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:12.016154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:12.016154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:12.016154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:12.016154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:12.016473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:12.016473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:12.016473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:12.016473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:12.016473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:12.016473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:12.016473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:12.016473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.016473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:12.017372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.017372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:12.017372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:12.017372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:12.017372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.017372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:12.017372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:12.017372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:47:12.017372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:12.017372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:12.017372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.017372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:12.017630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:12.017630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:12.017630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.017630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:12.017630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:12.017630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:12.017630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:12.017630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:12.017630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:12.017630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:12.017630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:12.017630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:12.017630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:12.017630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:12.017865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:12.017865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:12.017865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.017865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:12.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:12.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:12.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:12.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:12.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:12.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:12.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:12.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:12.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:12.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:12.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:12.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:12.019614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:12.019934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:12.019934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:12.019934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:12.019934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:12.019934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:12.019934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.019934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:12.033320 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:12.033639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:12.033767 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:12.033880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:12.034368 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:12.034762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.035357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:12.035357 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:12.035357 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:12.035357 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:12.035357 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:12.035357 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:12.035357 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.035357 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:12.035930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.035930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:12.035930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:12.035930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:12.035930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.035930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:12.035930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:12.035930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:12.035930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.035930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:12.035930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:12.035930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:12.035930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:12.035930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:12.035930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:12.036249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:12.036249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:12.036249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:12.036249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:12.036249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:12.036249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:12.036249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:12.036249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.036249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:12.036502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.036502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:12.036502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:12.036502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:12.036502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.036502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:12.036502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:12.036502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:12.036502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:12.036502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.036502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:12.036720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:12.036720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:12.036720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.036720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:12.036720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:12.036720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:12.036720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:12.036720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:12.036720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:12.036720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:12.036720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:12.036720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:12.036720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:12.036720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:12.036937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:12.036937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:12.036937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.036937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.037363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:12.037663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:12.037963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:12.037963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:12.037963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.037963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:12.266298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:12.266650 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:12.266760 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:12.266920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:12.268972 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:12.269167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:12.269310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.270233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:12.270233 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:12.270233 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:12.270233 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:12.270233 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:12.270233 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:12.270233 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.270233 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:12.271123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.271123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:12.271123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:12.271123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:12.271123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.271123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:12.271123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:12.271123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.271123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:12.271123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:12.271123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:12.271123 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.271408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:12.271408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:12.271408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:12.271408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:12.271408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:12.271408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:12.271408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:12.271408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:12.271408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:12.271408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:12.271408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:12.271408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:12.271408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:12.271664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.271664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:12.272395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.272395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:12.272395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:12.272395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:12.272395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:12.272395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:12.272395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.272395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:12.272395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:12.272395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.272395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:12.272395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:12.272395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.272395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:12.272395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:12.272395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:12.272745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:12.273053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:12.273053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.273053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:12.278714 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:12.280134 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:12.280250 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:12.280919 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:12.310954 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:12.311153 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.311676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:12.311676 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:12.311676 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:12.311676 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:12.311676 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:12.311676 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:12.311676 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.311676 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:12.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:12.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:12.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:12.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:12.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:12.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:12.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:12.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:12.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.312752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:12.312752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:12.312752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:12.312752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:12.312752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:12.312752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:12.312752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:12.312752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:12.312752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:12.312752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:12.312752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:12.312752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:12.312752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:12.313083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.313083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:12.313223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.313223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:12.313223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:12.313223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:12.313223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:12.313223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:12.313223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.313223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:12.313223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:12.313223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.313223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:12.313223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:12.313223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.313223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:12.313223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:12.313223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:12.313653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:12.314111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:12.314111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:12.314111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:16.950792 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:16.952007 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:16.952133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:16.954164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:16.954164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:16.954164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:16.954164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:16.954164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:16.954164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:16.954429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:16.954429 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:16.954429 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:16.954429 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:16.954429 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:16.954429 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:16.954429 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:16.954429 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:16.954970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:16.954970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:16.954970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:16.954970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:16.954970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:16.954970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:16.954970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:16.954970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:16.954970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:16.954970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:16.954970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:16.954970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:16.955458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:16.955458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:16.955458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:16.955458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:16.955458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:16.955458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:16.955458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:16.955458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:16.955458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:16.955458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:16.955458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:16.955458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:16.955458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:16.955815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:16.955815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:16.955815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:16.955815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:16.955815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:16.955815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:16.955815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:16.955815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:16.955815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:16.955815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:16.955815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:16.955815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:16.955815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:16.955815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:16.955815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:16.956362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:16.956915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:16.956915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:16.956915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:16.956915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:16.956915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:16.956915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:17.011678 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:17.012092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:17.014695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:17.014913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:47:17.016109 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:17.016109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.016891 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:17.016891 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:17.016891 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:17.016891 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:17.016891 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:17.016891 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:17.016891 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.016891 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:17.017673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.017673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:17.017673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:17.017673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:17.017673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.017673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:17.017673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:17.017673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:17.017673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.017673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:17.017673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:17.017673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:17.017673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:17.017673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:17.017673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:17.018108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:17.018108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:17.018108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:17.018108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:17.018108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:17.018108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:17.018108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:17.018108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.018108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:17.020637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.020637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:17.020637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:17.020637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:17.020637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.020637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:17.020637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:17.020637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:47:17.020637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:17.020637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:17.020637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.020637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:17.020928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:17.020928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:17.020928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.020928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:17.020928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:17.020928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:17.020928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:17.020928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:17.020928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:17.020928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:17.020928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:17.020928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:17.020928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:17.020928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:17.021180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:17.021180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:17.021180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.021180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:17.021814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.021814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:17.021814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:17.021814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:17.021814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:17.021814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:17.021814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.021814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:17.021814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:17.021814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.021814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:17.021814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:17.021814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.021814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:17.021814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:17.021814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:17.022163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:17.022498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:17.022498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:17.022498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:17.022498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:17.022498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:17.022498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.022498 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:17.033793 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:17.034066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:17.034173 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:17.034285 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:17.034811 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:17.035454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.035637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:17.035637 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:17.035637 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:17.035637 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:17.035637 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:17.035637 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:17.035637 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.035637 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:17.036258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.036258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:17.036258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:17.036258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:17.036258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.036258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:17.036258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:17.036258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:17.036258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.036258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:17.036258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:17.036258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:17.036258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:17.036258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:17.036258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:17.036616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:17.036616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:17.036616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:17.036616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:17.036616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:17.036616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:17.036616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:17.036616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.036616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:17.036928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.036928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:17.036928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:17.036928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:17.036928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.036928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:17.036928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:17.036928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:17.036928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:17.036928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.036928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:17.037135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:17.037135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:17.037135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.037135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:17.037135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:17.037135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:17.037135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:17.037135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:17.037135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:17.037135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:17.037135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:17.037135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:17.037135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:17.037135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:17.037386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:17.037386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:17.037386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.037386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.037672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:17.037980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:17.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:17.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:17.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:17.266710 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:17.266937 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:17.267036 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:17.267189 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:17.268402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:17.268402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:17.268402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.268564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:17.268564 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:17.268564 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:17.268564 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:17.268564 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:17.268564 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:17.268564 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.268564 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:17.269174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.269174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:17.269174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:17.269174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:17.269174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.269174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:17.269174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:17.269174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.269174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:17.269174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:17.269174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:17.269174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.269587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:17.269587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:17.269587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:17.269587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:17.269587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:17.269587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:17.269587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:17.269587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:17.269587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:17.269587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:17.269587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:17.269587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:17.269587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:17.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:17.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:17.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:17.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:17.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:17.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:17.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:17.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:17.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:17.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:17.269967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:17.270376 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:17.270909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:17.270909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:17.270909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:17.270909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:17.270909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.270909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:17.280559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:17.280768 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:17.280768 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:17.281326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:17.315662 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:17.315662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.315845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:17.315845 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:17.315845 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:17.315845 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:17.315845 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:17.315845 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:17.315845 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.315845 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:17.316677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.316677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:17.316677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:17.316677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:17.316677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.316677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:17.316677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:17.316677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.316677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:17.316677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:17.316677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:17.316677 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.316934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:17.316934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:17.316934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:17.316934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:17.316934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:17.316934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:17.316934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:17.316934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:17.316934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:17.316934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:17.316934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:17.316934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:17.316934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:17.317144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.317144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:17.317511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.317511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:17.317511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:17.317511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:17.317511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:17.317511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:17.317511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.317511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:17.317511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:17.317511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.317511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:17.317511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:17.317511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.317511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:17.317511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:17.317511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:17.317820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:17.318134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:17.318134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:17.318134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:21.952133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:21.952673 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:21.952673 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:21.953405 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:21.953704 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:21.953898 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:21.954068 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:21.954220 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:21.954355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:21.954941 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:21.954941 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:21.954941 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:21.954941 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:21.954941 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:21.954941 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:21.954941 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:21.954941 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:21.955646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:21.955646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:21.955646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:21.955646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:21.955646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:21.955646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:21.955646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:21.955646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:21.955646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:21.955646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:21.955646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:21.955646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:21.955904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:21.955904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:21.955904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:21.955904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:21.955904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:21.955904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:21.955904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:21.955904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:21.955904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:21.955904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:21.955904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:21.955904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:21.955904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:21.956136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:21.956136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:21.956318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:21.956318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:21.956318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:21.956318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:21.956318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:21.956318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:21.956318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:21.956318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:21.956318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:21.956318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:21.956318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:21.956318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:21.956318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:21.956318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:21.956318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:21.956318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:21.956747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:21.957129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:21.957129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:21.957129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:22.012172 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:22.012599 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:22.012693 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:22.012777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:47:22.013201 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:22.013401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.013912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:22.013912 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:22.013912 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:22.013912 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:22.013912 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:22.013912 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:22.013912 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.013912 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:22.014557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.014557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:22.014557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:22.014557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:22.014557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.014557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:22.014557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:22.014557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:22.014557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.014557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:22.014557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:22.014557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:22.014557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:22.014557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:22.014557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:22.014847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:22.014847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:22.014847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:22.014847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:22.014847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:22.014847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:22.014847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:22.014847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.014847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:22.015133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.015133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:22.015133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:22.015133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:22.015133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.015133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:22.015133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:22.015133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:47:22.015133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:22.015133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:22.015133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.015133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:22.015368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:22.015368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:22.015368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.015368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:22.015368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:22.015368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:22.015368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:22.015368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:22.015368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:22.015368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:22.015368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:22.015368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:22.015368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:22.015368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:22.015630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:22.015630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:22.015630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.015630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:22.015820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.015820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:22.015820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:22.015820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:22.015820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:22.015820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:22.015820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.015820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:22.015820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:22.015820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.015820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:22.015820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:22.015820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.015820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:22.015820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:22.015820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:22.016099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:22.016387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:22.016387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:22.016387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:22.016387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:22.016387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:22.016387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.016387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:22.034156 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:22.034552 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:22.034657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:22.034799 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:22.035226 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:22.035599 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.036178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:22.036178 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:22.036178 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:22.036178 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:22.036178 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:22.036178 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:22.036178 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.036178 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:22.036835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.036835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:22.036835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:22.036835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:22.036835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.036835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:22.036835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:22.036835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:22.036835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.036835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:22.036835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:22.036835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:22.036835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:22.036835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:22.036835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:22.037081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:22.037081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:22.037081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:22.037081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:22.037081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:22.037081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:22.037081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:22.037081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.037081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:22.037468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.037468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:22.037468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:22.037468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:22.037468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.037468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:22.037468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:22.037468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:22.037468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:22.037468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.037468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:22.037832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:22.037832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:22.037832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.037832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:22.037832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:22.037832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:22.037832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:22.037832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:22.037832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:22.037832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:22.037832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:22.037832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:22.037832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:22.037832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:22.038140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:22.038140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:22.038140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.038140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:22.038140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.038140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:22.038140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:22.038140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:22.038140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:22.038140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:22.038140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.038140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:22.038140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:22.038140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:22.038461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:22.038800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:22.038800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:22.038800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:22.038800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:22.038800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:22.038800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:22.038800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:22.038800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:22.038800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:22.038800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.038800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:22.267138 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:22.267523 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:22.267639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:22.267774 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:22.268218 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:22.268478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:22.268709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.269347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:22.269347 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:22.269347 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:22.269347 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:22.269347 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:22.269347 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:22.269347 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.269347 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:22.270424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.270424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:22.270424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:22.270424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:22.270424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.270424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:22.270424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:22.270424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.270424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:22.270424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:22.270424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:22.270424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.270696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:22.270696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:22.270696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:22.270696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:22.270696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:22.270696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:22.270696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:22.270696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:22.270696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:22.270696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:22.270696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:22.270696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:22.270696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:22.270916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.270916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:22.272093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.272093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:22.272093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:22.272093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:22.272093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:22.272093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:22.272093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.272093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:22.272093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:22.272093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.272093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:22.272093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:22.272093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.272093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:22.272093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:22.272093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:22.272398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:22.272726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:22.272726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.272726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:22.280808 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:22.281179 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:22.281267 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:22.281891 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:22.284194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=240 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:47:22.318586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:22.318902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.319601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:22.319601 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:22.319601 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:22.319601 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:22.319601 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:22.319601 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:22.319601 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.319601 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:22.320300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.320300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:22.320300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:22.320300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:22.320300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.320300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:22.320300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:22.320300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.320300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:22.320300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:22.320300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:22.320300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.320525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:22.320525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:22.320525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:22.320525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:22.320525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:22.320525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:22.320525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:22.320525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:22.320525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:22.320525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:22.320525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:22.320525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:22.320525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:22.320747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.320747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:22.321036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.321036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:22.321036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:22.321036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:22.321036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:22.321036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:22.321036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.321036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:22.321036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:22.321036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.321036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:22.321036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:22.321036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.321036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:22.321036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:22.321036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:22.321330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:22.321651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:22.321651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:22.321651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:26.952642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:26.953182 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:26.953182 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:26.953689 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:26.953894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:26.954022 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:26.954152 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:26.954263 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:26.954369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:26.954838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:26.954838 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:26.954838 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:26.954838 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:26.954838 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:26.954838 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:26.954838 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:26.954838 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:26.955526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:26.955526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:26.955526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:26.955526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:26.955526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:26.955526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:26.955526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:26.955526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:26.955526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:26.955526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:26.955526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:26.955526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:26.955851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:26.955851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:26.955851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:26.955851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:26.955851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:26.955851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:26.955851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:26.955851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:26.955851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:26.955851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:26.955851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:26.955851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:26.955851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:26.956161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:26.956161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:26.956278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:26.956278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:26.956278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:26.956278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:26.956278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:26.956278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:26.956278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:26.956278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:26.956278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:26.956278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:26.956278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:26.956278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:26.956278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:26.956278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:26.956278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:26.956278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:26.956707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:26.957222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:26.957222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:26.957222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:27.012819 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:27.012943 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:27.013010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:27.013123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:47:27.014244 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:27.014244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.014244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:27.014244 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:27.014244 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:27.014244 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:27.014244 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:27.014244 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:27.014244 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.014244 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:27.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:27.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:27.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:27.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:27.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:27.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:27.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:27.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:27.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:27.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:27.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:27.014885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:27.015124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:27.015124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:27.015124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:27.015124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:27.015124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:27.015124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:27.015124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:27.015124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.015124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:27.015412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.015412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:27.015412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:27.015412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:27.015412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.015412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:27.015412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:27.015412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:47:27.015412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:27.015412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:27.015412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.015412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:27.015656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:27.015656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:27.015656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.015656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:27.015656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:27.015656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:27.015656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:27.015656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:27.015656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:27.015656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:27.015656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:27.015656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:27.015656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:27.015656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:27.015867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:27.015867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:27.015867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.015867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:27.016111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.016111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:27.016111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:27.016111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:27.016111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:27.016111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:27.016111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.016111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:27.016111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:27.016111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.016111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:27.016111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:27.016111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.016111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:27.016111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:27.016111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:27.016394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:27.016726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:27.016726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:27.016726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:27.016726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:27.016726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:27.016726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.016726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:27.034549 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:27.034747 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:27.034832 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:27.034926 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:27.035588 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:27.035924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.036507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:27.036507 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:27.036507 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:27.036507 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:27.036507 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:27.036507 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:27.036507 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.036507 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:27.037103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.037103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:27.037103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:27.037103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:27.037103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.037103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:27.037103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:27.037103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:27.037103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.037103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:27.037103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:27.037103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:27.037103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:27.037103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:27.037103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:27.037345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:27.037345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:27.037345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:27.037345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:27.037345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:27.037345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:27.037345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:27.037345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.037345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:27.037721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.037721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:27.037721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:27.037721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:27.037721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.037721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:27.037721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:27.037721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:27.037721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:27.037721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.037721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:27.037909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:27.037909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:27.037909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.037909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:27.037909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:27.037909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:27.037909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:27.037909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:27.037909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:27.037909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:27.037909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:27.037909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:27.037909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:27.037909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:27.038139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:27.038139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:27.038139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.038139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.038386 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:27.038721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:27.039029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:27.039029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:27.039029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.039029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:27.267619 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:27.267850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:27.267950 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:27.268092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:27.268577 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:27.268778 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:27.268936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.269361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:27.269361 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:27.269361 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:27.269361 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:27.269361 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:27.269361 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:27.269361 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.269361 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:27.270043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.270043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:27.270043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:27.270043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:27.270043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.270043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:27.270043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:27.270043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.270043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:27.270043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:27.270043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:27.270043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.270370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:27.270370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:27.270370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:27.270370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:27.270370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:27.270370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:27.270370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:27.270370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:27.270370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:27.270370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:27.270370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:27.270370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:27.270370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:27.270728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.270728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:27.270828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.270828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:27.270828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:27.270828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:27.270828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:27.270828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:27.270828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.270828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:27.270828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:27.270828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.270828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:27.270828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:27.270828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.270828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:27.270828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:27.270828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:27.271551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:27.272049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:27.272049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.272049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:27.281244 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:27.281554 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:27.281660 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:27.282270 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:27.317500 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:27.317726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.318232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:27.318232 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:27.318232 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:27.318232 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:27.318232 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:27.318232 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:27.318232 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.318232 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:27.319070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.319070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:27.319070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:27.319070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:27.319070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.319070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:27.319070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:27.319070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.319070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:27.319070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:27.319070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:27.319070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.319403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:27.319403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:27.319403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:27.319403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:27.319403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:27.319403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:27.319403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:27.319403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:27.319403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:27.319403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:27.319403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:27.319403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:27.319403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:27.319746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.319746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:27.320127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.320127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:27.320127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:27.320127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:27.320127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:27.320127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:27.320127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.320127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:27.320127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:27.320127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.320127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:27.320127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:27.320127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.320127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:27.320127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:27.320127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:27.320556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:27.321112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:27.321112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:27.321112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:31.953057 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:31.953699 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:31.953699 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:31.954318 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:31.954615 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:31.954813 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:31.955002 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:31.955149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:31.955310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:31.955913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:31.955913 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:31.955913 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:31.955913 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:31.955913 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:31.955913 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:31.955913 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:31.955913 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:31.956679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:31.956679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:31.956679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:31.956679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:31.956679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:31.956679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:31.956679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:31.956679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:31.956679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:31.956679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:31.956679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:31.956679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:31.956932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:31.956932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:31.956932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:31.956932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:31.956932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:31.956932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:31.956932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:31.956932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:31.956932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:31.956932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:31.956932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:31.956932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:31.956932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:31.957154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:31.957154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:31.957425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:31.957425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:31.957425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:31.957425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:31.957425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:31.957425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:31.957425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:31.957425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:31.957425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:31.957425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:31.957425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:31.957425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:31.957425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:31.957425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:31.957425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:31.957425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:31.957750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:31.958092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:31.958092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:31.958092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:32.013191 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:32.013665 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:32.013822 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:32.013956 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:47:32.014488 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:32.014754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.015320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:32.015320 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:32.015320 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:32.015320 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:32.015320 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:32.015320 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:32.015320 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.015320 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:32.015968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.015968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:32.015968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:32.015968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:32.015968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.015968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:32.015968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:32.015968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:32.015968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.015968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:32.015968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:32.015968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:32.015968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:32.015968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:32.015968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:32.016420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:32.016420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:32.016420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:32.016420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:32.016420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:32.016420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:32.016420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:32.016420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.016420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:32.016708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.016708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:32.016708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:32.016708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:32.016708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.016708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:32.016708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:32.016708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:47:32.016708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:32.016708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:32.016708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.016708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:32.017015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:32.017015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:32.017015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.017015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:32.017015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:32.017015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:32.017015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:32.017015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:32.017015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:32.017015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:32.017015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:32.017015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:32.017015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:32.017015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:32.017367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:32.017367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:32.017367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.017367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:32.017367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.017367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:32.017367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:32.017367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:32.017367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:32.017367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:32.017367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.017367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:32.017367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:32.017367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:32.017764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:32.018248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:32.018248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:32.018248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:32.018248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:32.018248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:32.018248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:32.018248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:32.018248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:32.018248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:32.018248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:32.018248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.018248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:32.035026 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:32.035289 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:32.035407 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:32.035612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:32.035981 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:32.036227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.036999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:32.036999 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:32.036999 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:32.036999 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:32.036999 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:32.036999 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:32.036999 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.036999 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:32.037850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.037850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:32.037850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:32.037850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:32.037850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.037850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:32.037850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:32.037850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:32.037850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.037850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:32.037850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:32.037850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:32.037850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:32.037850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:32.037850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:32.038694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:32.038694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:32.038694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:32.038694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:32.038694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:32.038694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:32.038694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:32.038694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.038694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:32.038694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.038694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:32.038694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:32.038694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:32.038694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.039084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:32.039084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:32.039084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:32.039084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:32.039084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.039084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:32.039084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:32.039084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:32.039084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.039084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:32.039084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:32.039084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:32.039084 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:32.039423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:32.039423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:32.039423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:32.039423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:32.039423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:32.039423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:32.039423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:32.039423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:32.039423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:32.039423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.039423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:32.039423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:32.039778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.040245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:32.268002 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:32.268456 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:32.268554 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:32.268757 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:32.269212 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:32.269406 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:32.269598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.270022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:32.270022 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:32.270022 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:32.270022 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:32.270022 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:32.270022 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:32.270022 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.270022 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:32.270740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.270740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:32.270740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:32.270740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:32.270740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.270740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:32.270740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:32.270740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.270740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:32.270740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:32.270740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:32.270740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.271154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:32.271154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:32.271154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:32.271154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:32.271154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:32.271154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:32.271154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:32.271154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:32.271154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:32.271154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:32.271154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:32.271154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:32.271154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:32.271528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.271528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:32.271528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.271528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:32.271528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:32.271528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:32.271528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:32.271528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:32.271528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.271528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:32.271528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:32.271528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.271528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:32.271528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:32.271528 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:32.271955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:32.272431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:32.272431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:32.272431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:32.272431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:32.272431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.272431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:32.281700 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:32.282093 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:32.282192 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:32.282805 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:32.313745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:32.313994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.314679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:32.314679 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:32.314679 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:32.314679 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:32.314679 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:32.314679 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:32.314679 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.314679 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:32.315358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.033 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.315358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:32.315358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:32.315358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:32.315358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.315358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:32.315358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:32.315358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.315358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:32.315358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:32.315358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:32.315358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.315641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:32.315641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:32.315641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:32.315641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:32.315641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:32.315641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:32.315641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:32.315641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:32.315641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:32.315641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:32.315641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:32.315641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:32.315641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:32.315838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.315838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:32.316107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.316107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:32.316107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:32.316107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:32.316107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:32.316107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:32.316107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.316107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:32.316107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:32.316107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.316107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:32.316107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:32.316107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.316107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:32.316107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:32.316107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:32.316700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:32.317488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:32.317488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:32.317488 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:36.953430 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:36.953963 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:36.954067 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:36.954889 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:36.955085 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:36.955269 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:36.955416 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:36.955600 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:36.955736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:36.956225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:36.956225 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:36.956225 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:36.956225 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:36.956225 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:36.956225 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:36.956225 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:36.956225 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:36.957143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:36.957143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:36.957143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:36.957143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:36.957143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:36.957143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:36.957143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:36.957143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:36.957143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:36.957143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:36.957143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:36.957143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:36.957370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:36.957370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:36.957370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:36.957370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:36.957370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:36.957370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:36.957370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:36.957370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:36.957370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:36.957370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:36.957370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:36.957370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:36.957370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:36.957609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:36.957609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:36.958165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:36.958165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:36.958165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:36.958165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:36.958165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:36.958165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:36.958165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:36.958165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:36.958165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:36.958165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:36.958165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:36.958165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:36.958165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:36.958165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:36.958165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:36.958165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:36.958501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:36.958823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:36.958823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:36.958823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:37.013630 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:37.013940 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:37.014004 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:37.014125 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:47:37.014465 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:37.014671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.015079 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:37.015079 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:37.015079 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:37.015079 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:37.015079 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:37.015079 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:37.015079 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.015079 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:37.015839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.015839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:37.015839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:37.015839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:37.015839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.015839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:37.015839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:37.015839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:37.015839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.015839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:37.015839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:37.015839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:37.015839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:37.015839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:37.015839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:37.016271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:37.016271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:37.016271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:37.016271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:37.016271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:37.016271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:37.016271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:37.016271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.016271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:37.016671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.016671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:37.016671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:37.016671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:37.016671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.016671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:37.016671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:37.016671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:47:37.016671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:37.016671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:37.016671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.016671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:37.017031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:37.017031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:37.017031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.017031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:37.017031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:37.017031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:37.017031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:37.017031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:37.017031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:37.017031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:37.017031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:37.017031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:37.017031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:37.017031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:37.017433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:37.017433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:37.017433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.017433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:37.017679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.017679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:37.017679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:37.017679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:37.017679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:37.017679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:37.017679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.017679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:37.017679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:37.017679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.017679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:37.017679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:37.017679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.017679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:37.017679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:37.017679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:37.018137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:37.018686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:37.018686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:37.018686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:37.018686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:37.018686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:37.018686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.018686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:37.035375 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:37.035697 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:37.035810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:37.035938 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:37.036295 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:37.036577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.036994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:37.036994 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:37.036994 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:37.036994 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:37.036994 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:37.036994 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:37.036994 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.036994 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:37.037762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.037762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:37.037762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:37.037762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:37.037762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.037762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:37.037762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:37.037762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:37.037762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.037762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:37.037762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:37.037762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:37.037762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:37.037762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:37.037762 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:37.038078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:37.038078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:37.038078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:37.038078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:37.038078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:37.038078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:37.038078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:37.038078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.038078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:37.038643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.038643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:37.038643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:37.038643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:37.038643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.038643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:37.038643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:37.038643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:37.038643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:37.038643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.038643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:37.038841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:37.038841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:37.038841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.038841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:37.038841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:37.038841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:37.038841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:37.038841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:37.038841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:37.038841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:37.038841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:37.038841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:37.038841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:37.038841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:37.039131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:37.039131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:37.039131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.039131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.039795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:37.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:37.040378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:37.040378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:37.040378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.040378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:37.268500 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:37.268982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:37.269076 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:37.269221 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:37.269691 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:37.269904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:37.270089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.270594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:37.270594 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:37.270594 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:37.270594 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:37.270594 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:37.270594 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:37.270594 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.270594 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:37.271245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.271245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:37.271245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:37.271245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:37.271245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.271245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:37.271245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:37.271245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.271245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:37.271245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:37.271245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:37.271245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:37.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:37.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:37.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:37.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:37.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:37.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:37.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:37.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:37.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:37.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:37.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:37.271507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:37.271748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.271748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:37.271944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.271944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:37.271944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:37.271944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:37.271944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:37.271944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:37.271944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.271944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:37.271944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:37.271944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.271944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:37.271944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:37.271944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.271944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:37.271944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:37.271944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:37.272267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:37.272663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:37.272663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.272663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:37.282128 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:37.282422 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:37.282581 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:37.283275 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:37.313255 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:37.313444 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.313898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:37.313898 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:37.313898 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:37.313898 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:37.313898 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:37.313898 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:37.313898 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.313898 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:37.314560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.314560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:37.314560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:37.314560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:37.314560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.314560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:37.314560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:37.314560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.314560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:37.314560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:37.314560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:37.314560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:37.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:37.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:37.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:37.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:37.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:37.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:37.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:37.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:37.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:37.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:37.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:37.314853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:37.315073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.315073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:37.315329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.315329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:37.315329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:37.315329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:37.315329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:37.315329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:37.315329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.315329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:37.315329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:37.315329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.315329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:37.315329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:37.315329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.315329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:37.315329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:37.315329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:37.315642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:37.315962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:37.315962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:37.315962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:41.954033 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:41.954650 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:41.954650 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:41.955327 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:41.955587 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:41.955764 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:41.955934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:41.956108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:41.956255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:41.956953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:41.956953 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:41.956953 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:41.956953 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:41.956953 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:41.956953 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:41.956953 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:41.956953 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:41.957722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:41.957722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:41.957722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:41.957722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:41.957722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:41.957722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:41.957722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:41.957722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:41.957722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:41.957722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:41.957722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:41.957722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:41.958023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:41.958023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:41.958023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:41.958023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:41.958023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:41.958023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:41.958023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:41.958023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:41.958023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:41.958023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:41.958023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:41.958023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:41.958023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:41.958225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:41.958225 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:41.958460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:41.958460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:41.958460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:41.958460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:41.958460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:41.958460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:41.958460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:41.958460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:41.958460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:41.958460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:41.958460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:41.958460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:41.958460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:41.958460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:41.958460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:41.958460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:41.959168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:41.959502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:41.959502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:41.959502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:42.014015 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:42.014432 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:42.014593 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:42.014714 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:47:42.015181 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:42.015395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.016004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:42.016004 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:42.016004 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:42.016004 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:42.016004 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:42.016004 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:42.016004 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.016004 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:42.016839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.016839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:42.016839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:42.016839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:42.016839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.016839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:42.016839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:42.016839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:42.016839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.016839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:42.016839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:42.016839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:42.016839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:42.016839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:42.016839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:42.017088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:42.017088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:42.017088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:42.017088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:42.017088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:42.017088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:42.017088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:42.017088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.017088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:42.017682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.017682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:42.017682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:42.017682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:42.017682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.017682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:42.017682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:42.017682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:47:42.017682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:42.017682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:42.017682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.017682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:42.017882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:42.017882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:42.017882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.017882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:42.017882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:42.017882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:42.017882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:42.017882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:42.017882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:42.017882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:42.017882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:42.017882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:42.017882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:42.017882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:42.018095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:42.018095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:42.018095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.018095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:42.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:42.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:42.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:42.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:42.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:42.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:42.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:42.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:42.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:42.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:42.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:42.018646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:42.019162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:42.019654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:42.019654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:42.019654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:42.019654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:42.019654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:42.019654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.019654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:42.035763 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:42.036106 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:42.036233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:42.036335 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:42.036749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:42.037003 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.037550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:42.037550 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:42.037550 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:42.037550 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:42.037550 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:42.037550 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:42.037550 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.037550 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:42.038184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.038184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:42.038184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:42.038184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:42.038184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.038184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:42.038184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:42.038184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:42.038184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.038184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:42.038184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:42.038184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:42.038184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:42.038184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:42.038184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:42.038595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:42.038595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:42.038595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:42.038595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:42.038595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:42.038595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:42.038595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:42.038595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.038595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:42.038817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.038817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:42.038817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:42.038817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:42.038817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.038817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:42.038817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:42.038817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:42.038817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:42.038817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.038817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:42.039072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:42.039072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:42.039072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.039072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:42.039072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:42.039072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:42.039072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:42.039072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:42.039072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:42.039072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:42.039072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:42.039072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:42.039072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:42.039072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:42.039400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:42.039400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:42.039400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.039400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.040051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:42.041071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:42.041990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:42.041990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:42.041990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.041990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:42.268922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:42.269284 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:42.269390 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:42.269598 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:42.270014 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:42.270207 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:42.270361 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.270846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:42.270846 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:42.270846 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:42.270846 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:42.270846 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:42.270846 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:42.270846 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.270846 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:42.271508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.271508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:42.271508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:42.271508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:42.271508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.271508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:42.271508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:42.271508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.271508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:42.271508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:42.271508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:42.271508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.271925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:42.271925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:42.271925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:42.271925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:42.271925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:42.271925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:42.271925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:42.271925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:42.271925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:42.271925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:42.271925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:42.271925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:42.271925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:42.272159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.272159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:42.272159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.272159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:42.272159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:42.272159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:42.272159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:42.272159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:42.272159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.272159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:42.272159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:42.272159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.272159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:42.272159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:42.272159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:42.272429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:42.272822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:42.272822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:42.272822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:42.272822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:42.272822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.272822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:42.282525 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:42.282880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:42.282999 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:42.283664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:42.319030 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:42.319265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:42.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:42.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:42.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:42.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:42.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:42.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.319793 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:42.320619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.037 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.320619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:42.320619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:42.320619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:42.320619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.320619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:42.320619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:42.320619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.320619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:42.320619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:42.320619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:42.320619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.320944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:42.320944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:42.320944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:42.320944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:42.320944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:42.320944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:42.320944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:42.320944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:42.320944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:42.320944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:42.320944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:42.320944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:42.320944 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:42.321249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.321249 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:42.321643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.321643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:42.321643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:42.321643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:42.321643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:42.321643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:42.321643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.321643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:42.321643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:42.321643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.321643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:42.321643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:42.321643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.321643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:42.321643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:42.321643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:42.322659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:42.323256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:42.323256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:42.323256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:46.954458 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:46.955003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:46.955003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:46.955639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:46.955866 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:46.956102 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:46.956213 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:46.956343 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:46.956529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:46.957057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:46.957057 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:46.957057 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:46.957057 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:46.957057 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:46.957057 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:46.957057 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:46.957057 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:46.957808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:46.957808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:46.957808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:46.957808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:46.957808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:46.957808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:46.957808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:46.957808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:46.957808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:46.957808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:46.957808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:46.957808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:46.958039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:46.958039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:46.958039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:46.958039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:46.958039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:46.958039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:46.958039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:46.958039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:46.958039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:46.958039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:46.958039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:46.958039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:46.958039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:46.958234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:46.958234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:46.958521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:46.958521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:46.958521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:46.958521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:46.958521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:46.958521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:46.958521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:46.958521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:46.958521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:46.958521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:46.958521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:46.958521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:46.958521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:46.958521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:46.958521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:46.958521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:46.958812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:46.959535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:46.959535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:46.959535 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:47.014467 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:47.014744 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:47.014877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:47.015025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:47:47.015477 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:47.015678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.016152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:47.016152 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:47.016152 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:47.016152 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:47.016152 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:47.016152 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:47.016152 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.016152 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:47.016850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.016850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:47.016850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:47.016850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:47.016850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.016850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:47.016850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:47.016850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:47.016850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.016850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:47.016850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:47.016850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:47.016850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:47.016850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:47.016850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:47.017210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:47.017210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:47.017210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:47.017210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:47.017210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:47.017210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:47.017210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:47.017210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.017210 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:47.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:47.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:47.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:47.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:47.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:47.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:47:47.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:47.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:47.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:47.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:47.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:47.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:47.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:47.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:47.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:47.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:47.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:47.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:47.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:47.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:47.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:47.017703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:47.017959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:47.017959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:47.017959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.017959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:47.018053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.018053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:47.018053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:47.018053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:47.018053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:47.018053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:47.018053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.018053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:47.018053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:47.018053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.018053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:47.018053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:47.018053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.018053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:47.018053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:47.018053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:47.018380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:47.018752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:47.018752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:47.018752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:47.018752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:47.018752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:47.018752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.018752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:47.036115 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:47.036306 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:47.036431 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:47.036513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:47.037018 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:47.037205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.037726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:47.037726 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:47.037726 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:47.037726 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:47.037726 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:47.037726 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:47.037726 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.037726 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:47.038295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.038295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:47.038295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:47.038295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:47.038295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.038295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:47.038295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:47.038295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:47.038295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.038295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:47.038295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:47.038295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:47.038295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:47.038295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:47.038295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:47.038658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:47.038658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:47.038658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:47.038658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:47.038658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:47.038658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:47.038658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:47.038658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.038658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:47.038986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.038986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:47.038986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:47.038986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:47.038986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.038986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:47.038986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:47.038986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:47.038986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:47.038986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.038986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:47.039350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:47.039350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:47.039350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.039350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:47.039350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:47.039350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:47.039350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:47.039350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:47.039350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:47.039350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:47.039350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:47.039350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:47.039350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:47.039350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:47.039622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:47.039622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:47.039622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.039622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.039692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:47.039976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:47.040261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:47.040261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:47.040261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.040261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:47.269317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:47.269657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:47.269769 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:47.269905 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:47.270391 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:47.270632 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:47.270786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.271207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:47.271207 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:47.271207 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:47.271207 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:47.271207 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:47.271207 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:47.271207 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.271207 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:47.271943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.271943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:47.271943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:47.271943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:47.271943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.271943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:47.271943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:47.271943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.271943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:47.271943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:47.271943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:47.271943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.272308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:47.272308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:47.272308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:47.272308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:47.272308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:47.272308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:47.272308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:47.272308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:47.272308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:47.272308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:47.272308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:47.272308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:47.272308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:47.272581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.272581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:47.272646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.272646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:47.272646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:47.272646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:47.272646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:47.272646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:47.272646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.272646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:47.272646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:47.272646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.272646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:47.272646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:47.272646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.272646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:47.272646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:47.272646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:47.273036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:47.273412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:47.273412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.273412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:47.282847 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:47.283025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:47.283126 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:47.283683 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:47.309476 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:47.309689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.310186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:47.310186 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:47.310186 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:47.310186 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:47.310186 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:47.310186 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:47.310186 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.310186 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:47.310928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.310928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:47.310928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:47.310928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:47.310928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.310928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:47.310928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:47.310928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.310928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:47.310928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:47.310928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:47.310928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.311294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:47.311294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:47.311294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:47.311294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:47.311294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:47.311294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:47.311294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:47.311294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:47.311294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:47.311294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:47.311294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:47.311294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:47.311294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:47.311516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.311516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:47.311671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.311671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:47.311671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:47.311671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:47.311671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:47.311671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:47.311671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.311671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:47.311671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:47.311671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.311671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:47.311671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:47.311671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.311671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:47.311671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:47.311671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:47.311956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:47.312337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:47.312337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:47.312337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:51.955856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:51.956436 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:51.956436 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:51.957064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:51.957510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:51.957510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:51.957614 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:51.957740 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:51.957842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:51.958284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:51.958284 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:51.958284 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:51.958284 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:51.958284 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:51.958284 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:51.958284 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:51.958284 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:51.959104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:51.959104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:51.959104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:51.959104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:51.959104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:51.959104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:51.959104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:51.959104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:51.959104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:51.959104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:51.959104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:51.959104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:51.959494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:51.959494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:51.959494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:51.959494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:51.959494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:51.959494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:51.959494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:51.959494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:51.959494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:51.959494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:51.959494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:51.959494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:51.959494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:51.959828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:51.959828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:51.959893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:51.959893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:51.959893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:51.959893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:51.959893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:51.959893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:51.959893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:51.959893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:51.959893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:51.959893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:51.959893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:51.959893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:51.959893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:51.959893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:51.959893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:51.959893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:51.960285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:51.960792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:51.960792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:51.960792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:52.014945 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:52.015229 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:52.015360 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:52.015513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:47:52.015884 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:52.016077 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.016632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:52.016632 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:52.016632 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:52.016632 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:52.016632 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:52.016632 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:52.016632 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.016632 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:52.017375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.017375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:52.017375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:52.017375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:52.017375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.017375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:52.017375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:52.017375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:52.017375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.017375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:52.017375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:52.017375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:52.017375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:52.017375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:52.017375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:52.017721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:52.017721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:52.017721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:52.017721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:52.017721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:52.017721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:52.017721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:52.017721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.017721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:52.018254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.018254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:52.018254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:52.018254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:52.018254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.018254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:52.018254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:52.018254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:47:52.018254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:52.018254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:52.018254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.018254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:52.018551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:52.018551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:52.018551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.018551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:52.018551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:52.018551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:52.018551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:52.018551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:52.018551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:52.018551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:52.018551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:52.018551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:52.018551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:52.018551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:52.018810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:52.018810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:52.018810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.018810 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:52.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:52.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:52.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:52.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:52.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:52.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:52.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:52.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:52.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:52.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:52.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:52.019290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:52.019644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:52.019967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:52.019967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:52.019967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:52.019967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:52.019967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:52.019967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.019967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:52.036724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:52.037106 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:52.037254 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:52.038650 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:52.039097 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:52.039313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.039920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:52.039920 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:52.039920 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:52.039920 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:52.039920 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:52.039920 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:52.039920 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.039920 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:52.040530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.040530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:52.040530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:52.040530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:52.040530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.040530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:52.040530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:52.040530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:52.040530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.040530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:52.040530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:52.040530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:52.040530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:52.040530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:52.040530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:52.040874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:52.040874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:52.040874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:52.040874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:52.040874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:52.040874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:52.040874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:52.040874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.040874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:52.041260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.041260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:52.041260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:52.041260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:52.041260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.041260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:52.041260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:52.041260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:52.041260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:52.041260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.041260 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:52.041512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:52.041512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:52.041512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.041512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:52.041512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:52.041512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:52.041512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:52.041512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:52.041512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:52.041512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:52.041512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:52.041512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:52.041512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:52.041512 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:52.041785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:52.041785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:52.041785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.041785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.042159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:52.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:52.042862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:52.042862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:52.042862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.042862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:52.269757 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:52.270061 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:52.270188 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:52.270283 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:52.271612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:52.271612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:52.271612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.272050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:52.272050 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:52.272050 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:52.272050 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:52.272050 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:52.272050 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:52.272050 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.272050 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:52.272851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.272851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:52.272851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:52.272851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:52.272851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.272851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:52.272851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:52.272851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.272851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:52.272851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:52.272851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:52.272851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.273101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:52.273101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:52.273101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:52.273101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:52.273101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:52.273101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:52.273101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:52.273101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:52.273101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:52.273101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:52.273101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:52.273101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:52.273101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:52.273381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.273381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:52.273813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.273813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:52.273813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:52.273813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:52.273813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:52.273813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:52.273813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.273813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:52.273813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:52.273813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.273813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:52.273813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:52.273813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.273813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:52.273813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:52.273813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:52.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:52.274481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:52.274481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.274481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:52.283084 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:52.283276 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:52.283372 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:52.284026 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:52.310132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:52.310378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.311063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:52.311063 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:52.311063 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:52.311063 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:52.311063 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:52.311063 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:52.311063 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.311063 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:52.311961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.311961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:52.311961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:52.311961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:52.311961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.311961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:52.311961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:52.311961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.311961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:52.311961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:52.311961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:52.311961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:52.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:52.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:52.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:52.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:52.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:52.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:52.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:52.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:52.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:52.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:52.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:52.312370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:52.312794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.312794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:52.313070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.313070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:52.313070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:52.313070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:52.313070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:52.313070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:52.313070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.313070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:52.313070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:52.313070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.313070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:52.313070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:52.313070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.313070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:52.313070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:52.313070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:52.313418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:52.313804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:52.313804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:52.313804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:56.956254 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:56.957393 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:56.957393 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:56.957474 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:56.957610 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:56.957728 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:56.957832 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:56.957959 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:56.958059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:56.959777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:56.959777 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:56.959777 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:56.959777 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:56.959777 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:56.959777 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:56.959777 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:56.959777 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:56.959777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:56.959777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:56.959777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:56.959777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:56.959777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:56.959777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:56.959777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:56.960097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:56.960097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:56.960097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:56.960097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:56.960097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:56.960097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:56.960097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:56.960097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:56.960097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:56.960097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:56.960097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:56.960097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:56.960097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:56.960097 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:56.960346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:56.960346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:56.960346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:56.960346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:56.960346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:56.960346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:56.960346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:56.960346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:56.960346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:56.960346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:56.960346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:56.960346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:56.960346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:56.960346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:56.960646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:56.960991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:56.960991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:56.960991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:56.960991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:56.960991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:56.960991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:56.960991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:56.960991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:56.960991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:56.960991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:57.015302 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:57.015670 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:57.015781 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:57.015870 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:47:57.016193 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:57.016354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.016799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:57.016799 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:57.016799 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:57.016799 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:57.016799 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:57.016799 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:57.016799 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.016799 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:57.017325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.017325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:57.017325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:57.017325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:57.017325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.017325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:57.017325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:57.017325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:57.017325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.017325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:57.017325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:57.017325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:57.017325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:57.017325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:57.017325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:57.017653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:57.017653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:57.017653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:57.017653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:57.017653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:57.017653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:57.017653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:57.017653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.017653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:57.018061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.018061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:57.018061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:57.018061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:57.018061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.018061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:57.018061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:57.018061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:47:57.018061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:57.018061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:57.018061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.018061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:57.018280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:57.018280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:57.018280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.018280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:57.018280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:57.018280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:57.018280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:57.018280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:57.018280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:57.018280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:57.018280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:57.018280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:57.018280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:57.018280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:57.018581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:57.018581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:57.018581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.018581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:57.018826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.018826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:57.018826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:57.018826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:57.018826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:57.018826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:57.018826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.018826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:57.018826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:57.018826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.018826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:57.018826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:57.018826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.018826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:57.018826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:57.018826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:57.019092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:57.019393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:57.019393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:57.019393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:57.019393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:57.019393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:57.019393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.019393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:57.037167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:57.037313 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:57.037401 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:57.037547 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:57.037847 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:57.038030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.038383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:57.038383 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:57.038383 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:57.038383 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:57.038383 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:57.038383 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:57.038383 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.038383 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:57.038934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.038934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:47:57.038934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:57.038934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:57.038934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.038934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:57.038934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:57.038934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:47:57.038934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.038934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:47:57.038934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:57.038934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:47:57.038934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:57.038934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:47:57.038934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:47:57.039395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:57.039395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:47:57.039395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:57.039395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:47:57.039395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:47:57.039395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:57.039395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:47:57.039395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.039395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:47:57.039700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.039700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:57.039700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:57.039700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:57.039700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.039700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:57.039700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:57.039700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:57.039700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:57.039700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.039700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:57.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:57.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:57.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:57.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:57.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:57.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:57.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:57.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:57.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:57.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:57.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:57.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:57.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:57.040481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:57.040481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:57.040481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.040481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:57.040481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.040481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:57.040481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:57.040481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:57.040481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:57.040481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:57.040481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.040481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:57.040481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:57.040481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:57.040915 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:57.041468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:57.041468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:57.041468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:57.041468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:57.041468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:57.041468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:57.041468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:57.041468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:57.041468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:57.041468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.041468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:57.270147 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:57.270392 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:57.270550 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:57.270749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:57.271031 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:57.271234 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:57.271416 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.271893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:57.271893 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:57.271893 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:57.271893 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:57.271893 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:57.271893 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:57.271893 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.271893 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:57.272603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.272603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:57.272603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:57.272603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:57.272603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.272603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:57.272603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:57.272603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.272603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:57.272603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:57.272603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:57.272603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.272976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:57.272976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:57.272976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:57.272976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:57.272976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:57.272976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:57.272976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:57.272976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:57.272976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:57.272976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:57.272976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:57.272976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:57.272976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:57.273368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.273368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:57.273660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.273660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:57.273660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:57.273660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:57.273660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:57.273660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:57.273660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.273660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:57.273660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:57.273660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.273660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:57.273660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:57.273660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.273660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:57.273660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:57.273660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:57.273977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:57.274293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:57.274293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.274293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:47:57.283443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:47:57.283680 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:47:57.283776 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:47:57.284397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:47:57.313606 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:47:57.313728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.314141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:47:57.314141 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:57.314141 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:47:57.314141 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:47:57.314141 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:57.314141 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:47:57.314141 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.314141 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:47:57.314833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.314833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:47:57.314833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:57.314833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:47:57.314833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.314833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:57.314833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:57.314833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.314833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:57.314833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:57.314833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:47:57.314833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.315160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:47:57.315160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:57.315160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:47:57.315160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:57.315160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:47:57.315160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:47:57.315160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:57.315160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:47:57.315160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:57.315160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:47:57.315160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:47:57.315160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:57.315160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:47:57.315383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.315383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:47:57.315791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.315791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:47:57.315791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:47:57.315791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:47:57.315791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:47:57.315791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:47:57.315791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.315791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:47:57.315791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:47:57.315791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.315791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:47:57.315791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:47:57.315791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.315791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:47:57.315791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:47:57.315791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:47:57.316072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:47:57.316379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:47:57.316379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:47:57.316379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:01.956960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:01.957315 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:01.957315 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:01.957922 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:01.958059 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:01.958171 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f04ac03d-730f-4163-bed8-8911c59008e0, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:01.958271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DelStaticRoutesCommand(_result=None, lrouter=neutron-1def65b0-f351-4655-87fc-b825980c1fe7, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:01.958368 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:01.958477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:01.958894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:01.958894 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:01.958894 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:01.958894 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:01.958894 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:01.958894 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:01.958894 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:01.958894 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:01.959596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:01.959596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:01.959596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:01.959596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:01.959596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:01.959596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:01.959596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:01.959596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:01.959596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:01.959596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:01.959596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:01.959596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:01.959917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:01.959917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:01.959917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:01.959917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:01.959917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:01.959917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:01.959917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:01.959917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:01.959917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:01.959917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:01.959917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:01.959917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:01.959917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:01.960156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:01.960156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:01.960649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:01.960649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:01.960649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:01.960649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:01.960649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:01.960649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:01.960649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:01.960649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:01.960649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:01.960649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:01.960649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:01.960649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:01.960649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:01.960649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:01.960649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:01.960649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:01.961690 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:01.962043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:01.962043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:01.962043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:02.015927 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:02.016224 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:02.016224 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:02.016384 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:48:02.016795 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:02.016991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:02.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:02.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:02.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:02.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:02.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:02.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.017413 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:02.017986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.017986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:02.017986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:02.017986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:02.017986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.017986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:02.017986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:02.017986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:02.017986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.017986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:02.017986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:02.017986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:02.017986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:02.017986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:02.017986 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:02.018450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:02.018450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:02.018450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:02.018450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:02.018450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:02.018450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:02.018450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:02.018450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.018450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:02.018738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.018738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:02.018738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:02.018738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:02.018738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.018738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:02.018738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:02.018738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:48:02.018738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:02.018738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:02.018738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.018738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:02.019076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:02.019076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:02.019076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.019076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:02.019076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:02.019076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:02.019076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:02.019076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:02.019076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:02.019076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:02.019076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:02.019076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:02.019076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:02.019076 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:02.019485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:02.019485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:02.019485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.019485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:02.019485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.019485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:02.019485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:02.019485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:02.019485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:02.019485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:02.019485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.019485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:02.019485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:02.019485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:02.019890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:02.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:02.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:02.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:02.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:02.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:02.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:02.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:02.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:02.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:02.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:02.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:02.037653 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:02.037966 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:02.038074 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:02.038202 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:02.038647 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:02.038858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.039313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:02.039313 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:02.039313 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:02.039313 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:02.039313 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:02.039313 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:02.039313 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.039313 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:02.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:02.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:02.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:02.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:02.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:02.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:02.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:02.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:02.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:02.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:02.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:02.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:02.040295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:02.040295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:02.040295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:02.040295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:02.040295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:02.040295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:02.040295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:02.040295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.040295 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:02.040611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.040611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:02.040611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:02.040611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:02.040611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.040611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:02.040611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:02.040611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:02.040611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:02.040611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.040611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:02.040837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:02.040837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:02.040837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.040837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:02.040837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:02.040837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:02.040837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:02.040837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:02.040837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:02.040837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:02.040837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:02.040837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:02.040837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:02.040837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:02.041083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:02.041083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:02.041083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.041083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.041340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:02.041761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:02.042120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:02.042120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:02.042120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.042120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:02.270601 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:02.270836 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:02.270957 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:02.271092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:02.271361 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:02.271576 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:02.271743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.272134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:02.272134 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:02.272134 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:02.272134 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:02.272134 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:02.272134 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:02.272134 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.272134 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:02.272744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.272744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:02.272744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:02.272744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:02.272744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.272744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:02.272744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:02.272744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.272744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:02.272744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:02.272744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:02.272744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.273010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:02.273010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:02.273010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:02.273010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:02.273010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:02.273010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:02.273010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:02.273010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:02.273010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:02.273010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:02.273010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:02.273010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:02.273010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:02.273235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.273235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:02.273393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.273393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:02.273393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:02.273393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:02.273393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:02.273393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:02.273393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.273393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:02.273393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:02.273393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.273393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:02.273393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:02.273393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.273393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:02.273393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:02.273393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:02.273737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:02.274040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:02.274040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.274040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:02.283831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:02.284054 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:02.284149 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:02.284837 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:02.312555 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:02.312727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.313128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:02.313128 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:02.313128 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:02.313128 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:02.313128 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:02.313128 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:02.313128 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.313128 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:02.313772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.313772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:02.313772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:02.313772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:02.313772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.313772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:02.313772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:02.313772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.313772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:02.313772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:02.313772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:02.313772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.314015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:02.314015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:02.314015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:02.314015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:02.314015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:02.314015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:02.314015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:02.314015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:02.314015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:02.314015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:02.314015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:02.314015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:02.314015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:02.314228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.314228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:02.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:02.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:02.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:02.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:02.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:02.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:02.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:02.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:02.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:02.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:02.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:02.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:02.314742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:02.315073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:02.315073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:02.315073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:06.957347 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:06.957842 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:06.957842 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:06.958516 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:06.958771 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-3e9267f2-318f-43d1-b28b-efabfd85699b, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:06.958968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-311f3cda-f2b3-45b2-81de-0fea8df583d6, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:06.959115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:06.959756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:06.959756 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:06.959756 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:06.959756 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:06.959756 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:06.959756 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:06.959756 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:06.959756 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:06.960550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:06.960550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:06.960550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:06.960550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:06.960550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:06.960550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:06.960550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:06.960550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:06.960550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:06.960550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:06.960550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:06.960550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:06.961353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:06.961353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:06.961353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:06.961353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:06.961353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:06.961353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:06.961353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:06.961353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:06.961353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:06.961353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:06.961353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:06.961353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:06.961353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:06.961819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:06.961819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:06.961819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:06.961819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:06.961819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:06.961819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:06.961819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:06.961819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:06.961819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:06.961819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:06.961819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:06.961819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:06.961819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:06.961819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:06.961819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:06.962283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:06.964401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:06.964401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:06.964401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:06.964401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:06.964401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:06.964401 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:07.016262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:07.016639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:07.016761 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:07.016854 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:48:07.017219 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:07.017422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.017917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:07.017917 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:07.017917 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:07.017917 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:07.017917 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:07.017917 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:07.017917 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.017917 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:07.018572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.018572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:07.018572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:07.018572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:07.018572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.018572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:07.018572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:07.018572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:07.018572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.018572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:07.018572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:07.018572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:07.018572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:07.018572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:07.018572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:07.018990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:07.018990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:07.018990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:07.018990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:07.018990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:07.018990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:07.018990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:07.018990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.018990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:07.019248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.019248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:07.019248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:07.019248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:07.019248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.019248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:07.019248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:07.019248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:48:07.019248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:07.019248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:07.019248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.019248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:07.019650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:07.019650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:07.019650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.019650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:07.019650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:07.019650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:07.019650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:07.019650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:07.019650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:07.019650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:07.019650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:07.019650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:07.019650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:07.019650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:07.020009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:07.020009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:07.020009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.020009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:07.020009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.020009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:07.020009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:07.020009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:07.020009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:07.020009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:07.020009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.020009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:07.020009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:07.020009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:07.020406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:07.020983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:07.020983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:07.020983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:07.020983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:07.020983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:07.020983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:07.020983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:07.020983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:07.020983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:07.020983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:07.020983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.020983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:07.037988 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:07.038333 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:07.038333 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:07.038595 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:07.038790 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:07.038978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.039645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:07.039645 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:07.039645 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:07.039645 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:07.039645 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:07.039645 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:07.039645 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.039645 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:07.040021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.040021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:07.040021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:07.040021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:07.040021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.040021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:07.040021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:07.040021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:07.040021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.040021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:07.040021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:07.040021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:07.040021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:07.040021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:07.040021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:07.040415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:07.040415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:07.040415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:07.040415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:07.040415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:07.040415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:07.040415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:07.040415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.040415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:07.040710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.040710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:07.040710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:07.040710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:07.040710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.040710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:07.040710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:07.040710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:07.040710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:07.040710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.040710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:07.041250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:07.041250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:07.041250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.041250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.041320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:07.041630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:07.041910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:07.041910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:07.041910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.041910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:07.271013 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:07.271188 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:07.271288 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:07.271437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:07.271738 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:07.272493 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:07.272493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.272633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:07.272633 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:07.272633 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:07.272633 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:07.272633 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:07.272633 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:07.272633 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.272633 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:07.273191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.273191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:07.273191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:07.273191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:07.273191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.273191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:07.273191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:07.273191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.273191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:07.273191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:07.273191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:07.273191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.273412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:07.273412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:07.273412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:07.273412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:07.273412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:07.273412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:07.273412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:07.273412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:07.273412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:07.273412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:07.273412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:07.273412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:07.273412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:07.273649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.273649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:07.273907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.273907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:07.273907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:07.273907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:07.273907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:07.273907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:07.273907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.273907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:07.273907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:07.273907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.273907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:07.273907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:07.273907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.273907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:07.273907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:07.273907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:07.274188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:07.274646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:07.274646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.274646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:07.284211 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:07.284369 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:07.284453 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:07.285056 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:07.322663 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:07.322663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.322830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:07.322830 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:07.322830 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:07.322830 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:07.322830 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:07.322830 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:07.322830 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.322830 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:07.323362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.038 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.323362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:07.323362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:07.323362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:07.323362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.323362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:07.323362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:07.323362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.323362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:07.323362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:07.323362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:07.323362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.323655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:07.323655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:07.323655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:07.323655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:07.323655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:07.323655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:07.323655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:07.323655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:07.323655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:07.323655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:07.323655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:07.323655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:07.323655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:07.323880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.323880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:07.324050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.324050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:07.324050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:07.324050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:07.324050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:07.324050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:07.324050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.324050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:07.324050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:07.324050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.324050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:07.324050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:07.324050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.324050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:07.324050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:07.324050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:07.324353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:07.324726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:07.324726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:07.324726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:11.958210 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:11.959051 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:11.959142 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:11.959760 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:11.960248 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:11.960248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:11.960484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:11.960484 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:11.960484 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:11.960484 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:11.960484 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:11.960484 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:11.960484 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:11.960484 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:11.961179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:11.961179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:11.961179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:11.961179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:11.961179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:11.961179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:11.961179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:11.961179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:11.961179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:11.961179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:11.961179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:11.961179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:11.961600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:11.961600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:11.961600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:11.961600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:11.961600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:11.961600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:11.961600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:11.961600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:11.961600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:11.961600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:11.961600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:11.961600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:11.961600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:11.962070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:11.962070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:11.962070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:11.962070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:11.962070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:11.962070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:11.962070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:11.962070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:11.962070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:11.962070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:11.962070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:11.962070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:11.962070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:11.962070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:11.962070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:11.962506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:11.962989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:11.962989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:11.962989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:11.962989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:11.962989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:11.962989 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:12.016718 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:12.016998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:12.017113 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:12.017217 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:48:12.017698 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:12.017857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.018296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:12.018296 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:12.018296 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:12.018296 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:12.018296 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:12.018296 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:12.018296 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.018296 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:12.019112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.019112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:12.019112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:12.019112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:12.019112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.019112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:12.019112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:12.019112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:12.019112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.019112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:12.019112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:12.019112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:12.019112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:12.019112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:12.019112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:12.019408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:12.019408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:12.019408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:12.019408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:12.019408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:12.019408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:12.019408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:12.019408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.019408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:12.019814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.019814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:12.019814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:12.019814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:12.019814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.019814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:12.019814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:12.019814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:48:12.019814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:12.019814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:12.019814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.019814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:12.020025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:12.020025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:12.020025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.020025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:12.020025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:12.020025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:12.020025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:12.020025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:12.020025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:12.020025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:12.020025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:12.020025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:12.020025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:12.020025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:12.020233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:12.020233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:12.020233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.020233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:12.020678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.020678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:12.020678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:12.020678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:12.020678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:12.020678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:12.020678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.020678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:12.020678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:12.020678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.020678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:12.020678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:12.020678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.020678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:12.020678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:12.020678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:12.020950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:12.021223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:12.021223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:12.021223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:12.021223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:12.021223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:12.021223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.021223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:12.038382 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:12.038689 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:12.038798 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:12.038909 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:12.039228 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:12.039407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.039844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:12.039844 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:12.039844 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:12.039844 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:12.039844 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:12.039844 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:12.039844 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.039844 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:12.040501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.040501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:12.040501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:12.040501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:12.040501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.040501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:12.040501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:12.040501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:12.040501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.040501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:12.040501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:12.040501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:12.040501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:12.040501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:12.040501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:12.040789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:12.040789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:12.040789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:12.040789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:12.040789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:12.040789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:12.040789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:12.040789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.040789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:12.041588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.041588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:12.041588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:12.041588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:12.041588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.041588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:12.041588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:12.041588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:12.041588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:12.041588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.041588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:12.041795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:12.041795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:12.041795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.041795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:12.041795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:12.041795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:12.041795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:12.041795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:12.041795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:12.041795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:12.041795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:12.041795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:12.041795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:12.041795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:12.042021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:12.042021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:12.042021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.042021 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.042830 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:12.043141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:12.043464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:12.043464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:12.043464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.043464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:12.271397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:12.272021 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:12.272165 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:12.272946 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:12.273290 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:12.273890 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:12.273890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.274094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:12.274094 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:12.274094 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:12.274094 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:12.274094 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:12.274094 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:12.274094 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.274094 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:12.274828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.274828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:12.274828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:12.274828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:12.274828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.274828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:12.274828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:12.274828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.274828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:12.274828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:12.274828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:12.274828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.275102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:12.275102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:12.275102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:12.275102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:12.275102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:12.275102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:12.275102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:12.275102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:12.275102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:12.275102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:12.275102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:12.275102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:12.275102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:12.275331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.275331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:12.276105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.276105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:12.276105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:12.276105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:12.276105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:12.276105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:12.276105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.276105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:12.276105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:12.276105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.276105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:12.276105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:12.276105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.276105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:12.276105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:12.276105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:12.276423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:12.276792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:12.276792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.276792 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:12.284885 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:12.285201 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:12.285337 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:12.286096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:12.326815 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:12.327109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.327964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:12.327964 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:12.327964 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:12.327964 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:12.327964 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:12.327964 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:12.327964 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.327964 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:12.329103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.043 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.329103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:12.329103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:12.329103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:12.329103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.329103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:12.329103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:12.329103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.329103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:12.329103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:12.329103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:12.329103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.329529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:12.329529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:12.329529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:12.329529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:12.329529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:12.329529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:12.329529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:12.329529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:12.329529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:12.329529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:12.329529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:12.329529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:12.329529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:12.329910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.329910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:12.330191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.330191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:12.330191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:12.330191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:12.330191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:12.330191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:12.330191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.330191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:12.330191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:12.330191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.330191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:12.330191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:12.330191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.330191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:12.330191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:12.330191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:12.331089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:12.331419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:12.331419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:12.331419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:16.959146 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:16.959786 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:16.959786 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:16.960189 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:16.960393 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:16.960559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:16.960709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:16.961244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:16.961244 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:16.961244 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:16.961244 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:16.961244 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:16.961244 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:16.961244 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:16.961244 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:16.961949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:16.961949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:16.961949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:16.961949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:16.961949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:16.961949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:16.961949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:16.961949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:16.961949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:16.961949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:16.961949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:16.961949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:16.962430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:16.962430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:16.962430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:16.962430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:16.962430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:16.962430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:16.962430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:16.962430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:16.962430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:16.962430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:16.962430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:16.962430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:16.962430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:16.962942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:16.962942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:16.962942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:16.962942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:16.962942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:16.962942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:16.962942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:16.962942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:16.962942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:16.962942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:16.962942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:16.962942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:16.962942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:16.962942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:16.962942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:16.963289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:16.963715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:16.963715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:16.963715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:16.963715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:16.963715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:16.963715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:17.017039 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:17.017226 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:17.017325 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:17.017418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:48:17.018025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:17.018264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.018925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:17.018925 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:17.018925 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:17.018925 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:17.018925 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:17.018925 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:17.018925 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.018925 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:17.019492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.019492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:17.019492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:17.019492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:17.019492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.019492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:17.019492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:17.019492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:17.019492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.019492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:17.019492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:17.019492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:17.019492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:17.019492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:17.019492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:17.020006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:17.020006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:17.020006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:17.020006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:17.020006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:17.020006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:17.020006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:17.020006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.020006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:17.020283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.020283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:17.020283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:17.020283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:17.020283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.020283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:17.020283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:17.020283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:48:17.020283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:17.020283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:17.020283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.020283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:17.020664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:17.020664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:17.020664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.020664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:17.020664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:17.020664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:17.020664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:17.020664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:17.020664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:17.020664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:17.020664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:17.020664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:17.020664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:17.020664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:17.021044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:17.021044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:17.021044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.021044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:17.021044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.021044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:17.021044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:17.021044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:17.021044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:17.021044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:17.021044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.021044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:17.021044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:17.021044 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:17.021425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:17.021987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:17.021987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:17.021987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:17.021987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:17.021987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:17.021987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:17.021987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:17.021987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:17.021987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:17.021987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:17.021987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.021987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:17.038642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:17.038796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:17.038911 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:17.039040 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:17.039434 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:17.039729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.040200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:17.040200 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:17.040200 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:17.040200 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:17.040200 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:17.040200 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:17.040200 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.040200 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:17.041004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.041004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:17.041004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:17.041004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:17.041004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.041004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:17.041004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:17.041004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:17.041004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.041004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:17.041004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:17.041004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:17.041004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:17.041004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:17.041004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:17.041281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:17.041281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:17.041281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:17.041281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:17.041281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:17.041281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:17.041281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:17.041281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.041281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:17.041896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.041896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:17.041896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:17.041896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:17.041896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.041896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:17.041896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:17.041896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:17.041896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:17.041896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.041896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:17.042133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:17.042133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:17.042133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.042133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:17.042133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:17.042133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:17.042133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:17.042133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:17.042133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:17.042133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:17.042133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:17.042133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:17.042133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:17.042133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:17.042443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:17.042443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:17.042443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.042443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.043001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:17.043388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:17.043746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:17.043746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:17.043746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.043746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:17.272294 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:17.272768 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:17.272865 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:17.273152 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:17.275025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:17.275025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:17.275164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.276766 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:17.276766 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:17.276766 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:17.276766 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:17.276766 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:17.276766 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:17.276766 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.276766 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:17.277506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.004 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.277506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:17.277506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:17.277506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:17.277506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.277506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:17.277506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:17.277506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.277506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:17.277506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:17.277506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:17.277506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.277793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:17.277793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:17.277793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:17.277793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:17.277793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:17.277793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:17.277793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:17.277793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:17.277793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:17.277793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:17.277793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:17.277793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:17.277793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:17.278010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.278010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:17.278236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.278236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:17.278236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:17.278236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:17.278236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:17.278236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:17.278236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.278236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:17.278236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:17.278236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.278236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:17.278236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:17.278236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.278236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:17.278236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:17.278236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:17.281725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:17.282116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:17.282116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.282116 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:17.285233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:17.285602 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:17.285738 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:17.286347 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:17.312919 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:17.313115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.315859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:17.315859 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:17.315859 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:17.315859 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:17.315859 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:17.315859 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:17.315859 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.315859 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:17.315859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.315859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:17.315859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:17.315859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:17.315859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.315859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:17.315859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:17.316417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.316417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:17.316417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:17.316417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:17.316417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.316417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:17.316417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:17.316417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:17.316417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:17.316417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:17.316417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:17.316417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:17.316417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:17.316417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:17.316667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:17.316667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:17.316667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:17.316667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:17.316667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.316667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:17.316667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.316667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:17.316667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:17.316667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:17.316667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:17.316667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:17.316667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.316667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:17.316924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:17.317277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:17.317277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:17.317277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:17.317277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:17.317277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:17.317277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:17.317277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:17.317277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:17.317277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:21.959564 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:21.960064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:21.960108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:21.960676 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:21.960819 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:21.960951 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:21.961056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:21.961456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:21.961456 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:21.961456 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:21.961456 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:21.961456 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:21.961456 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:21.961456 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:21.961456 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:21.962120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:21.962120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:21.962120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:21.962120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:21.962120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:21.962120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:21.962120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:21.962120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:21.962120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:21.962120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:21.962120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:21.962120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:21.962372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:21.962372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:21.962372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:21.962372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:21.962372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:21.962372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:21.962372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:21.962372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:21.962372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:21.962372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:21.962372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:21.962372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:21.962372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:21.962628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:21.962628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:21.962834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:21.962834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:21.962834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:21.962834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:21.962834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:21.962834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:21.962834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:21.962834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:21.962834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:21.962834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:21.962834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:21.962834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:21.962834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:21.962834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:21.962834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:21.962834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:21.963121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:21.963458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:21.963458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:21.963458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:22.017531 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:22.017998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:22.018066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:22.018202 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:48:22.019010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:22.019284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.019935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:22.019935 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:22.019935 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:22.019935 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:22.019935 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:22.019935 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:22.019935 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.019935 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:22.020931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.020931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:22.020931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:22.020931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:22.020931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.020931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:22.020931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:22.020931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:22.020931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.020931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:22.020931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:22.020931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:22.020931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:22.020931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:22.020931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:22.021357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:22.021357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:22.021357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:22.021357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:22.021357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:22.021357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:22.021357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:22.021357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.021357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:22.021628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.021628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:22.021628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:22.021628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:22.021628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.021628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:22.021628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:22.021628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:48:22.021628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:22.021628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:22.021628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.021628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:22.021976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:22.021976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:22.021976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.021976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:22.021976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:22.021976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:22.021976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:22.021976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:22.021976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:22.021976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:22.021976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:22.021976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:22.021976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:22.021976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:22.022727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:22.022727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:22.022727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.022727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:22.022727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.022727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:22.022727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:22.022727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:22.022727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:22.022727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:22.022727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.022727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:22.022727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:22.022727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:22.022993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:22.023372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:22.023372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:22.023372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:22.023372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:22.023372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:22.023372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:22.023372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:22.023372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:22.023372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:22.023372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:22.023372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.023372 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:22.038971 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:22.039286 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:22.039353 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:22.039425 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:22.039960 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:22.040285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.040893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:22.040893 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:22.040893 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:22.040893 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:22.040893 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:22.040893 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:22.040893 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.040893 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:22.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:22.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:22.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:22.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:22.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:22.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:22.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:22.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:22.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:22.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:22.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:22.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:22.042245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:22.042245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:22.042245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:22.042245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:22.042245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:22.042245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:22.042245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:22.042245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.042245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:22.042598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.042598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:22.042598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:22.042598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:22.042598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.042598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:22.042598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:22.042598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:22.042598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:22.042598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.042598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:22.043029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:22.043029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:22.043029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.043029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:22.043029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:22.043029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:22.043029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:22.043029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:22.043029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:22.043029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:22.043029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:22.043029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:22.043029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:22.043029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:22.043427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:22.043427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:22.043427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.043427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.043673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:22.044198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:22.044684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:22.044684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:22.044684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.044684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:22.273018 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:22.273424 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:22.273637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:22.273871 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:22.274124 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:22.274319 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:22.274440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.274900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:22.274900 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:22.274900 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:22.274900 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:22.274900 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:22.274900 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:22.274900 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.274900 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:22.275556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.275556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:22.275556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:22.275556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:22.275556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.275556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:22.275556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:22.275556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.275556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:22.275556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:22.275556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:22.275556 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.275851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:22.275851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:22.275851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:22.275851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:22.275851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:22.275851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:22.275851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:22.275851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:22.275851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:22.275851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:22.275851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:22.275851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:22.275851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:22.276070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.276070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:22.276450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.276450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:22.276450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:22.276450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:22.276450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:22.276450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:22.276450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.276450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:22.276450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:22.276450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.276450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:22.276450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:22.276450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.276450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:22.276450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:22.276450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:22.276781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:22.277133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:22.277133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.277133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:22.285852 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:22.286128 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:22.286223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:22.286851 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:22.291275 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=241 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:48:22.313891 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:22.314031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:22.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:22.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:22.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:22.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:22.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:22.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.314449 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:22.315103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.315103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:22.315103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:22.315103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:22.315103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.315103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:22.315103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:22.315103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.315103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:22.315103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:22.315103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:22.315103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.315421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:22.315421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:22.315421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:22.315421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:22.315421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:22.315421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:22.315421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:22.315421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:22.315421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:22.315421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:22.315421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:22.315421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:22.315421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:22.315700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.315700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:22.315854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.315854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:22.315854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:22.315854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:22.315854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:22.315854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:22.315854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.315854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:22.315854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:22.315854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.315854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:22.315854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:22.315854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.315854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:22.315854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:22.315854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:22.316245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:22.316622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:22.316622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:22.316622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:26.960037 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:26.960617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:26.960617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:26.960966 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:26.961114 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:26.961224 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:26.961330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:26.961770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:26.961770 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:26.961770 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:26.961770 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:26.961770 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:26.961770 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:26.961770 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:26.961770 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:26.962377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:26.962377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:26.962377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:26.962377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:26.962377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:26.962377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:26.962377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:26.962377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:26.962377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:26.962377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:26.962377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:26.962377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:26.962674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:26.962674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:26.962674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:26.962674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:26.962674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:26.962674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:26.962674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:26.962674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:26.962674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:26.962674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:26.962674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:26.962674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:26.962674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:26.962917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:26.962917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:26.963135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:26.963135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:26.963135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:26.963135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:26.963135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:26.963135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:26.963135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:26.963135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:26.963135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:26.963135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:26.963135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:26.963135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:26.963135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:26.963135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:26.963135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:26.963135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:26.963617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:26.963996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:26.963996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:26.963996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:27.018042 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:27.018373 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:27.018499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:27.018630 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:48:27.018970 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:27.019126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.019510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:27.019510 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.019510 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:27.019510 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:27.019510 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.019510 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:27.019510 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.019510 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:27.020041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.020041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:27.020041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:27.020041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:27.020041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.020041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:27.020041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:27.020041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:27.020041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.020041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:27.020041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:27.020041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:27.020041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:27.020041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:27.020041 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:27.020348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:27.020348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:27.020348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.020348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:27.020348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:27.020348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.020348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:27.020348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.020348 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:27.020625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.020625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:27.020625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:27.020625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:27.020625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.020625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:27.020625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:27.020625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:48:27.020625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:27.020625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:27.020625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.020625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:27.020872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:27.020872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:27.020872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.020872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:27.020872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:27.020872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:27.020872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:27.020872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:27.020872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:27.020872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:27.020872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:27.020872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.020872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:27.020872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:27.021127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.021127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:27.021127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.021127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:27.021277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.021277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:27.021277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:27.021277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:27.021277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:27.021277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:27.021277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.021277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:27.021277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:27.021277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.021277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:27.021277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:27.021277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.021277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:27.021277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:27.021277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:27.021641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:27.022047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.022047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:27.022047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:27.022047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.022047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:27.022047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.022047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:27.039383 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:27.039770 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:27.039881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:27.039993 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:27.040318 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:27.040519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.040920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:27.040920 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.040920 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:27.040920 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:27.040920 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.040920 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:27.040920 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.040920 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:27.041409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.041409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:27.041409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:27.041409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:27.041409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.041409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:27.041409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:27.041409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:27.041409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.041409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:27.041409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:27.041409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:27.041409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:27.041409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:27.041409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:27.041713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:27.041713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:27.041713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.041713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:27.041713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:27.041713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.041713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:27.041713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.041713 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:27.042195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.042195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:27.042195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:27.042195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:27.042195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.042195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:27.042195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:27.042195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:27.042195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:27.042195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.042195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:27.042450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:27.042450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:27.042450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.042450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:27.042450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:27.042450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:27.042450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:27.042450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:27.042450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:27.042450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:27.042450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:27.042450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.042450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:27.042450 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:27.042715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.042715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:27.042715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.042715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.042919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:27.043281 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:27.043651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.043651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:27.043651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.043651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:27.093411 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:27.093613 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_for_inconsistencies {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:27.093727 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Checking Neutron and OVN revision consistency. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:27.099942 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Synchronizing Neutron and OVN databases started {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:440}} Jun 01 11:48:27.100059 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Number of inconsistencies found at create/update: networks=3 {{(pid=89427) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 01 11:48:27.100131 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Number of inconsistencies found at delete: router_ports=10 {{(pid=89427) _log /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:419}} Jun 01 11:48:27.100229 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 0d4134e5-3b3a-49d4-a466-c1b6f993b0ab (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:48:27.124477 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-0d4134e5-3b3a-49d4-a466-c1b6f993b0ab, resource={'id': '0d4134e5-3b3a-49d4-a466-c1b6f993b0ab', 'name': 'private', 'tenant_id': '43b6be50ae544caeb6a73eced65fb6a7', 'project_id': '43b6be50ae544caeb6a73eced65fb6a7', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['16527f37-b4ae-44e5-9382-abb5e59d1cad', 'a33d59fc-0fd4-45c5-9fff-82704a1922a6'], 'standard_attr_id': 12, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:23:10Z', 'updated_at': '2026-06-01T11:23:13Z', 'revision_number': 3, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 48058}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:27.124717 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-0d4134e5-3b3a-49d4-a466-c1b6f993b0ab, col_values=(('external_ids', {'neutron:network_name': 'private', 'neutron:mtu': '1372', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:27.124873 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-bedd3a6a-831e-4f6a-adf7-0842bd968bac, external_ids_update=None, columns={'tag': 48058}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:27.124974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.125482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:27.125482 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.125482 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:27.125482 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:27.125482 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.125482 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:27.125482 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.125482 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:27.126149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 0d4134e5-3b3a-49d4-a466-c1b6f993b0ab (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.126149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:27.126149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:48:27.126149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:48:27.126149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:48:27.126149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:48:27.126149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:48:27.126149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:27.126149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.126149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:27.126149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:27.126601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:27.126601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.126601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:27.126601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:27.126601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:27.126601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:27.126601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:27.126601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:27.126601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:27.126601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:27.126601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.126601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:27.126601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:27.126601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.127092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:27.127092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.127092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:27.127092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource c6c73189-2fc5-412b-9ed5-038efd7341bc (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:48:27.148064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, resource={'id': 'c6c73189-2fc5-412b-9ed5-038efd7341bc', 'name': 'public', 'tenant_id': '43b6be50ae544caeb6a73eced65fb6a7', 'project_id': '43b6be50ae544caeb6a73eced65fb6a7', 'admin_state_up': True, 'mtu': 1430, 'status': 'ACTIVE', 'subnets': ['21571d58-7621-44df-bdae-bad06600904d', '2bd944db-99d5-4573-8fc8-07e7942a1537'], 'standard_attr_id': 18, 'shared': False, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': True, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'is_default': True, 'tags': [], 'created_at': '2026-06-01T11:23:15Z', 'updated_at': '2026-06-01T11:23:22Z', 'revision_number': 3, 'provider:network_type': 'flat', 'provider:physical_network': 'public', 'provider:segmentation_id': None}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:27.148240 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-c6c73189-2fc5-412b-9ed5-038efd7341bc, col_values=(('external_ids', {'neutron:network_name': 'public', 'neutron:mtu': '1430', 'neutron:revision_number': '3', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'flat', 'neutron:provnet-physical-network': 'public'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false', 'fdb_age_threshold': '300', 'broadcast-arps-to-all-routers': 'true'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:27.148457 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-4ecb0f57-2a36-41bc-88d1-5cf4f555b097, external_ids_update=None, columns={'tag': []}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:27.148708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.150057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:27.150057 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.150057 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:27.150057 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:27.150057 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.150057 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:27.150057 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.150057 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:27.150057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource c6c73189-2fc5-412b-9ed5-038efd7341bc (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.150057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:27.150057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:48:27.150057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:48:27.150057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:48:27.150057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:48:27.150456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:48:27.150456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:27.150456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.150456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:27.150456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:27.150456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:27.150456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.150456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:27.150456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:27.150456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:27.150456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:27.150456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:27.150456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:27.150764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:27.150764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:27.150764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.150764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:27.150764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:27.150764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.150764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:27.150764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.150764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:27.150764 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 19da64d5-62e3-4b38-84f1-7d0971806ce9 (type: networks) at create/update {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:449}} Jun 01 11:48:27.173297 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(_result=None, name=neutron-19da64d5-62e3-4b38-84f1-7d0971806ce9, resource={'id': '19da64d5-62e3-4b38-84f1-7d0971806ce9', 'name': 'shared', 'tenant_id': '43b6be50ae544caeb6a73eced65fb6a7', 'project_id': '43b6be50ae544caeb6a73eced65fb6a7', 'admin_state_up': True, 'mtu': 1372, 'status': 'ACTIVE', 'subnets': ['c855657f-8ba2-422e-88fd-c5a71ec03ed7'], 'standard_attr_id': 26, 'shared': True, 'availability_zone_hints': [], 'availability_zones': [], 'ipv4_address_scope': None, 'ipv6_address_scope': None, 'router:external': False, 'qinq': False, 'vlan_transparent': False, 'description': '', 'port_security_enabled': True, 'tags': [], 'created_at': '2026-06-01T11:24:23Z', 'updated_at': '2026-06-01T11:24:23Z', 'revision_number': 2, 'provider:network_type': 'geneve', 'provider:physical_network': None, 'provider:segmentation_id': 19987}, resource_type=networks, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:27.173380 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(_result=None, table=Logical_Switch, record=neutron-19da64d5-62e3-4b38-84f1-7d0971806ce9, col_values=(('external_ids', {'neutron:network_name': 'shared', 'neutron:mtu': '1372', 'neutron:revision_number': '2', 'neutron:availability_zone_hints': '', 'neutron:provnet-network-type': 'geneve'}), ('other_config', {'mcast_snoop': 'false', 'mcast_flood_unregistered': 'false', 'vlan-passthru': 'false'})), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:27.173633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): SetLSwitchPortCommand(_result=None, lport=provnet-cb9dd134-19dd-491b-aafb-e4edb81fcf60, external_ids_update=None, columns={'tag': 19987}, if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:27.173707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.174128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:27.174128 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.174128 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:27.174128 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:27.174128 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.174128 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:27.174128 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.174128 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:27.174871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Failed to fix resource 19da64d5-62e3-4b38-84f1-7d0971806ce9 (type: networks): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.174871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:27.174871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 462, in check_for_inconsistencies Jun 01 11:48:27.174871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row) Jun 01 11:48:27.174871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 370, in _fix_create_update Jun 01 11:48:27.174871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_update'](context, n_obj) Jun 01 11:48:27.174871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2334, in update_network Jun 01 11:48:27.174871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:27.174871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.174871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:27.174871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:27.175080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:27.175080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.175080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:27.175080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:27.175080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:27.175080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:27.175080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:27.175080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:27.175080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:27.175080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:27.175080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.175080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:27.175080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:27.175080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.175329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:27.175329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.175329 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:27.175329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 05e73b9a-1af7-4f42-b8ae-0d3d7873b442 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:48:27.175329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(05e73b9a-1af7-4f42-b8ae-0d3d7873b442) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:48:27.179477 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 2cea6e44-6ac0-49de-97be-2553592819c9 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:48:27.179652 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(2cea6e44-6ac0-49de-97be-2553592819c9) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:48:27.183321 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 3a464d17-eb60-48ed-9aca-74e66e5467a4 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:48:27.183509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(3a464d17-eb60-48ed-9aca-74e66e5467a4) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:48:27.186972 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 47714e6d-1962-4188-ad04-982f39dc7cf7 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:48:27.187156 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(47714e6d-1962-4188-ad04-982f39dc7cf7) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:48:27.190572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 4c6b3cf2-e705-4d91-a405-aea4ec4b1d91 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:48:27.190767 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(4c6b3cf2-e705-4d91-a405-aea4ec4b1d91) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:48:27.194255 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 64c1bc42-615e-4bd7-b6c0-b6e62f628072 (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:48:27.194436 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(64c1bc42-615e-4bd7-b6c0-b6e62f628072) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:48:27.197737 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 68751736-4c5c-471b-b0eb-a13a1fc0c15f (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:48:27.198118 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(68751736-4c5c-471b-b0eb-a13a1fc0c15f) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:48:27.202682 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 8ebf307f-bc43-47dc-8c7d-a88d420f49bf (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:48:27.202872 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(8ebf307f-bc43-47dc-8c7d-a88d420f49bf) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:48:27.207433 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 90f34f8c-667e-45a8-a346-47368c275fdf (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:48:27.207678 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(90f34f8c-667e-45a8-a346-47368c275fdf) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:48:27.211356 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Maintenance task: Fixing resource 9e0207ff-4677-4c9d-bc24-f5d5d6ca9f0d (type: router_ports) at delete {{(pid=89427) check_for_inconsistencies /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:471}} Jun 01 11:48:27.211613 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.db.ovn_revision_numbers_db [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] delete_revision(9e0207ff-4677-4c9d-bc24-f5d5d6ca9f0d) {{(pid=89427) delete_revision /opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py:112}} Jun 01 11:48:27.215955 np0000001841 neutron-ovn-maintenance-worker[89427]: INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_for_inconsistencies finished in 0.122 seconds Jun 01 11:48:27.273404 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:27.273689 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:27.273801 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:27.273970 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:27.274258 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:27.274438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:27.274603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.274993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:27.274993 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.274993 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:27.274993 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:27.274993 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.274993 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:27.274993 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.274993 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:27.275731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.275731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:27.275731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:27.275731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:27.275731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.275731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:27.275731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:27.275731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.275731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:27.275731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:27.275731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:27.275731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.275984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:27.275984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:27.275984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:27.275984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:27.275984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:27.275984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:27.275984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:27.275984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:27.275984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.275984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:27.275984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:27.275984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.275984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:27.276299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.276299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:27.276612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.276612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:27.276612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:27.276612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:27.276612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:27.276612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:27.276612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.276612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:27.276612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:27.276612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.276612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:27.276612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:27.276612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.276612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:27.276612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:27.276612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:27.277096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.277640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:27.277640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.277640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:27.286174 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:27.286546 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:27.286658 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:27.287223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:27.317326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:27.317508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.317938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:27.317938 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.317938 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:27.317938 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:27.317938 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.317938 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:27.317938 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.317938 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:27.318623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.318623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:27.318623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:27.318623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:27.318623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.318623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:27.318623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:27.318623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.318623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:27.318623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:27.318623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:27.318623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.319000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:27.319000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:27.319000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:27.319000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:27.319000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:27.319000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:27.319000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:27.319000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:27.319000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.319000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:27.319000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:27.319000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.319000 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:27.319353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.319353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:27.319353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.319353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:27.319353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:27.319353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:27.319353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:27.319353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:27.319353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.319353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:27.319353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:27.319353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.319353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:27.319353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:27.319353 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:27.319807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:27.320318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:27.320318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:27.320318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:27.320318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:27.320318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:27.320318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:31.960431 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:31.960981 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:31.960981 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:31.961426 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:31.961611 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:31.961720 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:31.961831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:31.962236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:31.962236 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:31.962236 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:31.962236 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:31.962236 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:31.962236 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:31.962236 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:31.962236 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:31.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:31.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:31.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:31.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:31.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:31.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:31.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:31.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:31.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:31.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:31.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:31.962926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:31.963183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:31.963183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:31.963183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:31.963183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:31.963183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:31.963183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:31.963183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:31.963183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:31.963183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:31.963183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:31.963183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:31.963183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:31.963183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:31.963422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:31.963422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:31.963668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:31.963668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:31.963668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:31.963668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:31.963668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:31.963668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:31.963668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:31.963668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:31.963668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:31.963668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:31.963668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:31.963668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:31.963668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:31.963668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:31.963668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:31.963668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:31.963953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:31.964268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:31.964268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:31.964268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:32.018409 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:32.018785 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:32.018903 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:32.018964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:48:32.019277 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:32.019433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.019852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:32.019852 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:32.019852 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:32.019852 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:32.019852 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:32.019852 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:32.019852 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.019852 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:32.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:32.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:32.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:32.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:32.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:32.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:32.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:32.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:32.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:32.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:32.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:32.020364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:32.020805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:32.020805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:32.020805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:32.020805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:32.020805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:32.020805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:32.020805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:32.020805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.020805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:32.021064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.021064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:32.021064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:32.021064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:32.021064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.021064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:32.021064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:32.021064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:48:32.021064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:32.021064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:32.021064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.021064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:32.021411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:32.021411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:32.021411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.021411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:32.021411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:32.021411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:32.021411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:32.021411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:32.021411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:32.021411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:32.021411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:32.021411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:32.021411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:32.021411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:32.021837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:32.021837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:32.021837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.021837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:32.021956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.021956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:32.021956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:32.021956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:32.021956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:32.021956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:32.021956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.021956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:32.021956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:32.021956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.021956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:32.021956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:32.021956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.021956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:32.021956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:32.021956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:32.022379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:32.022881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:32.022881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:32.022881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:32.022881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:32.022881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:32.022881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.022881 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:32.039729 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:32.039973 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:32.040081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:32.040197 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:32.040509 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:32.040719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.041049 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:32.041049 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:32.041049 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:32.041049 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:32.041049 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:32.041049 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:32.041049 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.041049 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:32.041526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.041526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:32.041526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:32.041526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:32.041526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.041526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:32.041526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:32.041526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:32.041526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.041526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:32.041526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:32.041526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:32.041526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:32.041526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:32.041526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:32.041954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:32.041954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:32.041954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:32.041954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:32.041954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:32.041954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:32.041954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:32.041954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.041954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:32.042222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.042222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:32.042222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:32.042222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:32.042222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.042222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:32.042222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:32.042222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:32.042222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:32.042222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.042222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:32.042562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:32.042562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:32.042562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.042562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:32.042562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:32.042562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:32.042562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:32.042562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:32.042562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:32.042562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:32.042562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:32.042562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:32.042562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:32.042562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:32.042957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:32.042957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:32.042957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.042957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.043170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:32.043683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:32.044200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:32.044200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:32.044200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.044200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:32.273821 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:32.274177 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:32.274307 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:32.274478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:32.275161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:32.275431 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:32.275658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.276524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:32.276524 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:32.276524 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:32.276524 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:32.276524 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:32.276524 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:32.276524 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.276524 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:32.277358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.277358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:32.277358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:32.277358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:32.277358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.277358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:32.277358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:32.277358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.277358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:32.277358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:32.277358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:32.277358 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:32.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:32.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:32.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:32.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:32.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:32.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:32.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:32.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:32.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:32.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:32.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:32.277627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:32.277884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.277884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:32.279228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.279228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:32.279228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:32.279228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:32.279228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:32.279228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:32.279228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.279228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:32.279228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:32.279228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.279228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:32.279228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:32.279228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.279228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:32.279228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:32.279228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:32.279658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:32.280029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:32.280029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.280029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:32.286708 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:32.288839 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:32.288968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:32.290448 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:32.318274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:32.318274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.318513 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:32.318513 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:32.318513 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:32.318513 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:32.318513 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:32.318513 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:32.318513 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.318513 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:32.319364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.319364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:32.319364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:32.319364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:32.319364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.319364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:32.319364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:32.319364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.319364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:32.319364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:32.319364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:32.319364 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.319633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:32.319633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:32.319633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:32.319633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:32.319633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:32.319633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:32.319633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:32.319633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:32.319633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:32.319633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:32.319633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:32.319633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:32.319633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:32.319869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.319869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:32.320496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.320496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:32.320496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:32.320496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:32.320496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:32.320496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:32.320496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.320496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:32.320496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:32.320496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.320496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:32.320496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:32.320496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.320496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:32.320496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:32.320496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:32.320836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:32.321164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:32.321164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:32.321164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:36.960920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:36.961432 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:36.961611 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:36.962324 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:36.962535 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:36.962712 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:36.962835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:36.963345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:36.963345 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:36.963345 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:36.963345 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:36.963345 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:36.963345 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:36.963345 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:36.963345 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:36.964402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:36.964402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:36.964402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:36.964402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:36.964402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:36.964402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:36.964402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:36.964402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:36.964402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:36.964402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:36.964402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:36.964402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:36.965132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:36.965132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:36.965132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:36.965132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:36.965132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:36.965132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:36.965132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:36.965132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:36.965132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:36.965132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:36.965132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:36.965132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:36.965132 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:36.965848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:36.965848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:36.965848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:36.965848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:36.965848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:36.965848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:36.965848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:36.965848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:36.965848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:36.965848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:36.965848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:36.965848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:36.965848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:36.965848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:36.965848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:36.966247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:36.966655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:36.966655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:36.966655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:36.966655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:36.966655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:36.966655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:37.018920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:37.019364 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:37.019513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:37.019645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:48:37.020061 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:37.020267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.020784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:37.020784 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:37.020784 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:37.020784 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:37.020784 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:37.020784 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:37.020784 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.020784 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:37.021624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.021624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:37.021624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:37.021624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:37.021624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.021624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:37.021624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:37.021624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:37.021624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.021624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:37.021624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:37.021624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:37.021624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:37.021624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:37.021624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:37.021920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:37.021920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:37.021920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:37.021920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:37.021920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:37.021920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:37.021920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:37.021920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.021920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:37.022408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.022408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:37.022408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:37.022408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:37.022408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.022408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:37.022408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:37.022408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:48:37.022408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:37.022408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:37.022408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.022408 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:37.022674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:37.022674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:37.022674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.022674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:37.022674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:37.022674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:37.022674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:37.022674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:37.022674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:37.022674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:37.022674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:37.022674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:37.022674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:37.022674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:37.022897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:37.022897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:37.022897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.022897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:37.023431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.023431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:37.023431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:37.023431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:37.023431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:37.023431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:37.023431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.023431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:37.023431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:37.023431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.023431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:37.023431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:37.023431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.023431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:37.023431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:37.023431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:37.023734 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:37.024024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:37.024024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:37.024024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:37.024024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:37.024024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:37.024024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.024024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:37.040162 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:37.040643 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:37.040751 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:37.040870 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:37.041271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:37.041514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.042026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:37.042026 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:37.042026 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:37.042026 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:37.042026 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:37.042026 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:37.042026 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.042026 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:37.042875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.042875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:37.042875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:37.042875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:37.042875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.042875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:37.042875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:37.042875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:37.042875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.042875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:37.042875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:37.042875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:37.042875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:37.042875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:37.042875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:37.043140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:37.043140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:37.043140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:37.043140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:37.043140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:37.043140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:37.043140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:37.043140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.043140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:37.043756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.043756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:37.043756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:37.043756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:37.043756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.043756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:37.043756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:37.043756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:37.043756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:37.043756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.043756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:37.043975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:37.043975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:37.043975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.043975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:37.043975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:37.043975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:37.043975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:37.043975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:37.043975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:37.043975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:37.043975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:37.043975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:37.043975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:37.043975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:37.044196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:37.044196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:37.044196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.044196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.044608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:37.044913 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:37.045214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:37.045214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:37.045214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.045214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:37.274236 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:37.274499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:37.274656 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:37.274780 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:37.275122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:37.275369 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:37.275614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.276095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:37.276095 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:37.276095 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:37.276095 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:37.276095 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:37.276095 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:37.276095 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.276095 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:37.276736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.276736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:37.276736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:37.276736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:37.276736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.276736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:37.276736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:37.276736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.276736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:37.276736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:37.276736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:37.276736 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.276985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:37.276985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:37.276985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:37.276985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:37.276985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:37.276985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:37.276985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:37.276985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:37.276985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:37.276985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:37.276985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:37.276985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:37.276985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:37.277223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.277223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:37.277502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.277502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:37.277502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:37.277502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:37.277502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:37.277502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:37.277502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.277502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:37.277502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:37.277502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.277502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:37.277502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:37.277502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.277502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:37.277502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:37.277502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:37.278375 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:37.278816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:37.278816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.278816 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:37.288886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:37.289123 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:37.289232 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:37.289883 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:37.316290 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:37.316568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.316956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:37.316956 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:37.316956 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:37.316956 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:37.316956 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:37.316956 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:37.316956 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.316956 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:37.317683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.317683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:37.317683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:37.317683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:37.317683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.317683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:37.317683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:37.317683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.317683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:37.317683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:37.317683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:37.317683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.317965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:37.317965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:37.317965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:37.317965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:37.317965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:37.317965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:37.317965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:37.317965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:37.317965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:37.317965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:37.317965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:37.317965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:37.317965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:37.318168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.318168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:37.318860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.318860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:37.318860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:37.318860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:37.318860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:37.318860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:37.318860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.318860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:37.318860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:37.318860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.318860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:37.318860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:37.318860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.318860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:37.318860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:37.318860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:37.319208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:37.319529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:37.319529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:37.319529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:41.961524 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:41.962008 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:41.962008 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:41.962430 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:41.962628 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:41.962753 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:41.962836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:41.963224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:41.963224 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:41.963224 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:41.963224 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:41.963224 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:41.963224 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:41.963224 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:41.963224 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:41.963864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:41.963864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:41.963864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:41.963864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:41.963864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:41.963864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:41.963864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:41.963864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:41.963864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:41.963864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:41.963864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:41.963864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:41.964100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:41.964100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:41.964100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:41.964100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:41.964100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:41.964100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:41.964100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:41.964100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:41.964100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:41.964100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:41.964100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:41.964100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:41.964100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:41.964299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:41.964299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:41.964593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:41.964593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:41.964593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:41.964593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:41.964593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:41.964593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:41.964593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:41.964593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:41.964593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:41.964593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:41.964593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:41.964593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:41.964593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:41.964593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:41.964593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:41.964593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:41.964870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:41.965150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:41.965150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:41.965150 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:42.019438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:42.019762 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:42.019871 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:42.019975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:48:42.020306 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:42.020460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.020889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:42.020889 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:42.020889 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:42.020889 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:42.020889 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:42.020889 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:42.020889 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.020889 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:42.021396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.021396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:42.021396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:42.021396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:42.021396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.021396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:42.021396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:42.021396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:42.021396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.021396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:42.021396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:42.021396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:42.021396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:42.021396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:42.021396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:42.021709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:42.021709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:42.021709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:42.021709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:42.021709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:42.021709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:42.021709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:42.021709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.021709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:42.022046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.022046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:42.022046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:42.022046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:42.022046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.022046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:42.022046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:42.022046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:48:42.022046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:42.022046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:42.022046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.022046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:42.022527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:42.022527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:42.022527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.022527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:42.022527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:42.022527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:42.022527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:42.022527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:42.022527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:42.022527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:42.022527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:42.022527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:42.022527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:42.022527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:42.023075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:42.023075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:42.023075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.023075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:42.023075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.023075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:42.023075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:42.023075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:42.023075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:42.023075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:42.023075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.023075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:42.023075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:42.023075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:42.023299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:42.023719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:42.023719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:42.023719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:42.023719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:42.023719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:42.023719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:42.023719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:42.023719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:42.023719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:42.023719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:42.023719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.023719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:42.040678 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:42.040909 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:42.041000 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:42.041111 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:42.041401 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:42.041619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.041960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:42.041960 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:42.041960 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:42.041960 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:42.041960 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:42.041960 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:42.041960 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.041960 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:42.042439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.042439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:42.042439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:42.042439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:42.042439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.042439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:42.042439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:42.042439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:42.042439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.042439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:42.042439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:42.042439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:42.042439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:42.042439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:42.042439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:42.042733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:42.042733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:42.042733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:42.042733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:42.042733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:42.042733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:42.042733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:42.042733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.042733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:42.043078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.043078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:42.043078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:42.043078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:42.043078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.043078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:42.043078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:42.043078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:42.043078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:42.043078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.043078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:42.043475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:42.043475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:42.043475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.043475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:42.043475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:42.043475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:42.043475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:42.043475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:42.043475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:42.043475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:42.043475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:42.043475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:42.043475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:42.043475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:42.044059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:42.044059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:42.044059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.044059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:42.044059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.044059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:42.044059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:42.044059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:42.044059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:42.044059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:42.044059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.044059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:42.044059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:42.044059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:42.044351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:42.045038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:42.045038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:42.045038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:42.045038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:42.045038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:42.045038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:42.045038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:42.045038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:42.045038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:42.045038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.045038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:42.274795 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:42.275041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:42.275182 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:42.275409 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:42.275767 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:42.275958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:42.276085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.276530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:42.276530 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:42.276530 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:42.276530 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:42.276530 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:42.276530 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:42.276530 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.276530 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:42.277475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.277475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:42.277475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:42.277475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:42.277475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.277475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:42.277475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:42.277475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.277475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:42.277475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:42.277475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:42.277475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.277883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:42.277883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:42.277883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:42.277883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:42.277883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:42.277883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:42.277883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:42.277883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:42.277883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:42.277883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:42.277883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:42.277883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:42.277883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:42.278213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.278213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:42.278783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.278783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:42.278783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:42.278783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:42.278783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:42.278783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:42.278783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.278783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:42.278783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:42.278783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.278783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:42.278783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:42.278783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.278783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:42.278783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:42.278783 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:42.279214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:42.279706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:42.279706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.279706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:42.289371 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:42.289648 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:42.289800 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:42.290679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:42.324044 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:42.324292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.324882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:42.324882 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:42.324882 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:42.324882 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:42.324882 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:42.324882 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:42.324882 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.324882 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:42.325557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.325557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:42.325557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:42.325557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:42.325557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.325557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:42.325557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:42.325557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.325557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:42.325557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:42.325557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:42.325557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.325907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:42.325907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:42.325907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:42.325907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:42.325907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:42.325907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:42.325907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:42.325907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:42.325907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:42.325907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:42.325907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:42.325907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:42.325907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:42.326232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.326232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:42.326232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.326232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:42.326232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:42.326232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:42.326232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:42.326232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:42.326232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.326232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:42.326232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:42.326232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.326232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:42.326232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:42.326232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:42.326662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:42.327159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:42.327159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:42.327159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:42.327159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:42.327159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:42.327159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:46.961879 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:46.962369 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:46.962369 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:46.962894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:46.963041 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:46.963183 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:46.963291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:46.963751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:46.963751 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:46.963751 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:46.963751 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:46.963751 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:46.963751 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:46.963751 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:46.963751 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:46.964347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:46.964347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:46.964347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:46.964347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:46.964347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:46.964347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:46.964347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:46.964347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:46.964347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:46.964347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:46.964347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:46.964347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:46.964647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:46.964647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:46.964647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:46.964647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:46.964647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:46.964647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:46.964647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:46.964647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:46.964647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:46.964647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:46.964647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:46.964647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:46.964647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:46.964879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:46.964879 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:46.965331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:46.965331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:46.965331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:46.965331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:46.965331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:46.965331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:46.965331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:46.965331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:46.965331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:46.965331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:46.965331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:46.965331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:46.965331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:46.965331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:46.965331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:46.965331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:46.966231 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:46.966829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:46.966829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:46.966829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:47.019881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:47.020175 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:47.020267 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:47.020377 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:48:47.020744 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:47.020900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.021309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:47.021309 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:47.021309 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:47.021309 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:47.021309 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:47.021309 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:47.021309 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.021309 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:47.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:47.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:47.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:47.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:47.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:47.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:47.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:47.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:47.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:47.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:47.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:47.021851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:47.022242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:47.022242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:47.022242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:47.022242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:47.022242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:47.022242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:47.022242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:47.022242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.022242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:47.022557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.022557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:47.022557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:47.022557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:47.022557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.022557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:47.022557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:47.022557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:48:47.022557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:47.022557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:47.022557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.022557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:47.022823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:47.022823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:47.022823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.022823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:47.022823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:47.022823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:47.022823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:47.022823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:47.022823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:47.022823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:47.022823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:47.022823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:47.022823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:47.022823 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:47.023043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:47.023043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:47.023043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.023043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:47.023043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.023043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:47.023043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:47.023043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:47.023043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:47.023043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:47.023043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.023043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:47.023043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:47.023043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:47.023274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:47.023621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:47.023621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:47.023621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:47.023621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:47.023621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:47.023621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:47.023621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:47.023621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:47.023621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:47.023621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:47.023621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.023621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:47.041157 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:47.041472 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:47.041676 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:47.041842 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:47.042190 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:47.042374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.042831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:47.042831 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:47.042831 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:47.042831 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:47.042831 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:47.042831 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:47.042831 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.042831 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:47.043621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.043621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:47.043621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:47.043621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:47.043621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.043621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:47.043621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:47.043621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:47.043621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.043621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:47.043621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:47.043621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:47.043621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:47.043621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:47.043621 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:47.044043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:47.044043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:47.044043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:47.044043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:47.044043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:47.044043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:47.044043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:47.044043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.044043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:47.044601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.044601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:47.044601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:47.044601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:47.044601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.044601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:47.044601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:47.044601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:47.044601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:47.044601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.044601 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:47.045440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:47.045440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:47.045440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.045440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:47.045440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:47.045440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:47.045440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:47.045440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:47.045440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:47.045440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:47.045440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:47.045440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:47.045440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:47.045440 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:47.046204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:47.046204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:47.046204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.046204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:47.046204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.046204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:47.046204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:47.046204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:47.046204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:47.046204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:47.046204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.046204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:47.046204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:47.046204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:47.047107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:47.048051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:47.048051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:47.048051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:47.048051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:47.048051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:47.048051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:47.048051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:47.048051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:47.048051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:47.048051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.048051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:47.275194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:47.275495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:47.275640 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:47.275794 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:47.276117 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:47.276302 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:47.276425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.276878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:47.276878 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:47.276878 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:47.276878 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:47.276878 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:47.276878 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:47.276878 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.276878 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:47.277483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.277483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:47.277483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:47.277483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:47.277483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.277483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:47.277483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:47.277483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.277483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:47.277483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:47.277483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:47.277483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.277751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:47.277751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:47.277751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:47.277751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:47.277751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:47.277751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:47.277751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:47.277751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:47.277751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:47.277751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:47.277751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:47.277751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:47.277751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:47.277955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.277955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:47.278301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.278301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:47.278301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:47.278301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:47.278301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:47.278301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:47.278301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.278301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:47.278301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:47.278301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.278301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:47.278301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:47.278301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.278301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:47.278301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:47.278301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:47.279140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:47.279681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:47.279681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.279681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:47.289778 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:47.290069 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:47.290132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:47.290800 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:47.318035 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:47.318190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.318624 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:47.318624 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:47.318624 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:47.318624 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:47.318624 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:47.318624 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:47.318624 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.318624 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:47.319228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.319228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:47.319228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:47.319228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:47.319228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.319228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:47.319228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:47.319228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.319228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:47.319228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:47.319228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:47.319228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.319466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:47.319466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:47.319466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:47.319466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:47.319466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:47.319466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:47.319466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:47.319466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:47.319466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:47.319466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:47.319466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:47.319466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:47.319466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:47.319719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.319719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:47.320129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.320129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:47.320129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:47.320129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:47.320129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:47.320129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:47.320129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.320129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:47.320129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:47.320129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.320129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:47.320129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:47.320129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.320129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:47.320129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:47.320129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:47.320609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:47.320909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:47.320909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:47.320909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:51.962316 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:51.962842 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:51.962842 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:51.963424 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:51.963640 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:51.963755 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:51.963867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:51.964320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:51.964320 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:51.964320 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:51.964320 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:51.964320 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:51.964320 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:51.964320 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:51.964320 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:51.965012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:51.965012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:51.965012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:51.965012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:51.965012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:51.965012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:51.965012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:51.965012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:51.965012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:51.965012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:51.965012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:51.965012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:51.965263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:51.965263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:51.965263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:51.965263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:51.965263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:51.965263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:51.965263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:51.965263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:51.965263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:51.965263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:51.965263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:51.965263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:51.965263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:51.965485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:51.965485 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:51.965758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:51.965758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:51.965758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:51.965758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:51.965758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:51.965758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:51.965758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:51.965758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:51.965758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:51.965758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:51.965758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:51.965758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:51.965758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:51.965758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:51.965758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:51.965758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:51.966048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:51.966583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:51.966583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:51.966583 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:52.020308 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:52.020688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:52.020825 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:52.020962 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:48:52.021439 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:52.021671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.022146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:52.022146 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:52.022146 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:52.022146 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:52.022146 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:52.022146 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:52.022146 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.022146 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:52.022948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.022948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:52.022948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:52.022948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:52.022948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.022948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:52.022948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:52.022948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:52.022948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.022948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:52.022948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:52.022948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:52.022948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:52.022948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:52.022948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:52.023428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:52.023428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:52.023428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:52.023428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:52.023428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:52.023428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:52.023428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:52.023428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.023428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:52.023694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.023694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:52.023694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:52.023694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:52.023694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.023694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:52.023694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:52.023694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:48:52.023694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:52.023694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:52.023694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.023694 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:52.023952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:52.023952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:52.023952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.023952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:52.023952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:52.023952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:52.023952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:52.023952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:52.023952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:52.023952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:52.023952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:52.023952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:52.023952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:52.023952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:52.024240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:52.024240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:52.024240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.024240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:52.024669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.024669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:52.024669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:52.024669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:52.024669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:52.024669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:52.024669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.024669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:52.024669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:52.024669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.024669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:52.024669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:52.024669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.024669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:52.024669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:52.024669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:52.024956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:52.025291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:52.025291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:52.025291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:52.025291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:52.025291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:52.025291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.025291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:52.041531 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:52.042064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:52.042160 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:52.042299 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:52.042850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:52.043059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.043533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:52.043533 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:52.043533 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:52.043533 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:52.043533 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:52.043533 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:52.043533 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.043533 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:52.044134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.044134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:52.044134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:52.044134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:52.044134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.044134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:52.044134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:52.044134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:52.044134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.044134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:52.044134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:52.044134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:52.044134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:52.044134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:52.044134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:52.044785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:52.044785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:52.044785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:52.044785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:52.044785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:52.044785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:52.044785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:52.044785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.044785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:52.044785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.044785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:52.044785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:52.044785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:52.044785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.045014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:52.045014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:52.045014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:52.045014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:52.045014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.045014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:52.045014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:52.045014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:52.045014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.045014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:52.045014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:52.045014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:52.045014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:52.045235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:52.045235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:52.045235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:52.045235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:52.045235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:52.045235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:52.045235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:52.045235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:52.045235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:52.045235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.045235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.045610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:52.046104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:52.046883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:52.046883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:52.046883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.046883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:52.275702 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:52.275965 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:52.276074 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:52.276221 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:52.276524 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:52.276748 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:52.276871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.277244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:52.277244 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:52.277244 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:52.277244 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:52.277244 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:52.277244 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:52.277244 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.277244 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:52.277839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.277839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:52.277839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:52.277839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:52.277839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.277839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:52.277839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:52.277839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.277839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:52.277839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:52.277839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:52.277839 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.278099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:52.278099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:52.278099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:52.278099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:52.278099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:52.278099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:52.278099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:52.278099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:52.278099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:52.278099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:52.278099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:52.278099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:52.278099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:52.278309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.278309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:52.278609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.278609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:52.278609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:52.278609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:52.278609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:52.278609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:52.278609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.278609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:52.278609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:52.278609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.278609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:52.278609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:52.278609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.278609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:52.278609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:52.278609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:52.279126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:52.279752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:52.279752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.279752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:52.290127 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:52.290488 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:52.290669 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:52.291397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:52.319980 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:52.320147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.320505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:52.320505 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:52.320505 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:52.320505 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:52.320505 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:52.320505 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:52.320505 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.320505 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:52.321174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.321174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:52.321174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:52.321174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:52.321174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.321174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:52.321174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:52.321174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.321174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:52.321174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:52.321174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:52.321174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.321586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:52.321586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:52.321586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:52.321586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:52.321586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:52.321586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:52.321586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:52.321586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:52.321586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:52.321586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:52.321586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:52.321586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:52.321586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:52.322139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.322139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:52.322139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.322139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:52.322139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:52.322139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:52.322139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:52.322139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:52.322139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.322139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:52.322139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:52.322139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.322139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:52.322139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:52.322139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:52.322817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:52.323213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:52.323213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:52.323213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:52.323213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:52.323213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:52.323213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:56.962777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:56.963288 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:56.963288 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:56.963864 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:56.964108 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:56.964279 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:56.964447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:56.965055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:56.965055 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:56.965055 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:56.965055 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:56.965055 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:56.965055 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:56.965055 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:56.965055 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:56.965787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:56.965787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:56.965787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:56.965787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:56.965787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:56.965787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:56.965787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:56.965787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:56.965787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:56.965787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:56.965787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:56.965787 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:56.966039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:56.966039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:56.966039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:56.966039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:56.966039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:56.966039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:56.966039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:56.966039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:56.966039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:56.966039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:56.966039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:56.966039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:56.966039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:56.966264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:56.966264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:56.966579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:56.966579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:56.966579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:56.966579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:56.966579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:56.966579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:56.966579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:56.966579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:56.966579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:56.966579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:56.966579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:56.966579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:56.966579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:56.966579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:56.966579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:56.966579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:56.966898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:56.967219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:56.967219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:56.967219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:57.020681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:57.020933 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:57.021063 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:57.021161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:48:57.021677 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:57.021859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:57.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:57.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:57.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:57.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:57.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:57.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.022307 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:57.022926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.022926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:57.022926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:57.022926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:57.022926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.022926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:57.022926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:57.022926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:57.022926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.022926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:57.022926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:57.022926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:57.022926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:57.022926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:57.022926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:57.023202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:57.023202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:57.023202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:57.023202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:57.023202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:57.023202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:57.023202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:57.023202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.023202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:57.023620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.023620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:57.023620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:57.023620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:57.023620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.023620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:57.023620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:57.023620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:48:57.023620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:57.023620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:57.023620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.023620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:57.024100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:57.024100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:57.024100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.024100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:57.024100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:57.024100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:57.024100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:57.024100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:57.024100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:57.024100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:57.024100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:57.024100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:57.024100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:57.024100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:57.024461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:57.024461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:57.024461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.024461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:57.024461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.024461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:57.024461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:57.024461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:57.024461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:57.024461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:57.024461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.024461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:57.024461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:57.024461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:57.024725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:57.025013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:57.025013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:57.025013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:57.025013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:57.025013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:57.025013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:57.025013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:57.025013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:57.025013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:57.025013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:57.025013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.025013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:57.042013 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:57.042210 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:57.042287 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:57.042388 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:57.042861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:57.043081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.043517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:57.043517 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:57.043517 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:57.043517 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:57.043517 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:57.043517 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:57.043517 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.043517 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:57.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:48:57.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:57.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:57.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:57.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:57.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:48:57.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:48:57.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:57.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:48:57.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:57.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:48:57.044031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:48:57.044304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:57.044304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:48:57.044304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:57.044304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:48:57.044304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:48:57.044304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:57.044304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:48:57.044304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.044304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:48:57.044653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.044653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:57.044653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:57.044653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:57.044653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.044653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:57.044653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:57.044653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:57.044653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:57.044653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.044653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:57.045213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:57.045213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:57.045213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.045213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:57.045213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:57.045213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:57.045213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:57.045213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:57.045213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:57.045213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:57.045213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:57.045213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:57.045213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:57.045213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:57.045740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:57.045740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:57.045740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.045740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:57.045740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.045740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:57.045740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:57.045740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:57.045740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:57.045740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:57.045740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.045740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:57.045740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:57.045740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:57.045960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:57.046258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:57.046258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:57.046258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:57.046258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:57.046258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:57.046258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:57.046258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:57.046258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:57.046258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:57.046258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.046258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:57.276025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:57.276280 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:57.276379 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:57.276533 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:57.276848 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:57.277036 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:57.277160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.277587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:57.277587 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:57.277587 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:57.277587 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:57.277587 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:57.277587 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:57.277587 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.277587 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:57.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:57.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:57.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:57.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:57.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:57.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:57.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:57.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:57.278182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.278431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:57.278431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:57.278431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:57.278431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:57.278431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:57.278431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:57.278431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:57.278431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:57.278431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:57.278431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:57.278431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:57.278431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:57.278431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:57.278655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.278655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:57.278888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.278888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:57.278888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:57.278888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:57.278888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:57.278888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:57.278888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.278888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:57.278888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:57.278888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.278888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:57.278888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:57.278888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.278888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:57.278888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:57.278888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:57.279428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:57.280082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:57.280082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.280082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:48:57.290408 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:48:57.290623 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:48:57.290709 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:48:57.291277 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:48:57.316250 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:48:57.316464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.317010 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:48:57.317010 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:57.317010 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:48:57.317010 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:48:57.317010 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:57.317010 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:48:57.317010 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.317010 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:48:57.317662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.317662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:48:57.317662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:57.317662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:48:57.317662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.317662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:57.317662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:57.317662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.317662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:57.317662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:57.317662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:48:57.317662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.317897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:48:57.317897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:57.317897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:48:57.317897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:57.317897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:48:57.317897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:48:57.317897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:57.317897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:48:57.317897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:57.317897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:48:57.317897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:48:57.317897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:57.317897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:48:57.318092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.318092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:48:57.318330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.318330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:48:57.318330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:48:57.318330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:48:57.318330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:48:57.318330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:48:57.318330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.318330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:48:57.318330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:48:57.318330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.318330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:48:57.318330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:48:57.318330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.318330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:48:57.318330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:48:57.318330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:48:57.318629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:48:57.318916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:48:57.318916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:48:57.318916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:01.963185 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:01.963919 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:01.963919 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:01.964436 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:01.964795 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:01.965039 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:01.965223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:01.967141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:01.967141 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:01.967141 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:01.967141 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:01.967141 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:01.967141 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:01.967141 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:01.967141 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:01.967141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:01.967141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:01.967141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:01.967141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:01.967141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:01.967141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:01.967141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:01.967509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:01.967509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:01.967509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:01.967509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:01.967509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:01.967509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:01.967509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:01.967509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:01.967509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:01.967509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:01.967509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:01.967509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:01.967509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:01.967509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:01.967745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:01.967745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:01.967745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:01.967745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:01.967745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:01.967745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:01.967745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:01.967745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:01.967745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:01.967745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:01.967745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:01.967745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:01.967745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:01.967745 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:01.967977 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:01.968319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:01.968319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:01.968319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:01.968319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:01.968319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:01.968319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:01.968319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:01.968319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:01.968319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:01.968319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:02.021130 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:02.021597 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:02.021597 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:02.021597 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:49:02.022120 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:02.022452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.023042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:02.023042 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:02.023042 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:02.023042 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:02.023042 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:02.023042 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:02.023042 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.023042 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:02.023685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.023685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:02.023685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:02.023685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:02.023685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.023685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:02.023685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:02.023685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:02.023685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.023685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:02.023685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:02.023685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:02.023685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:02.023685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:02.023685 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:02.023972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:02.023972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:02.023972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:02.023972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:02.023972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:02.023972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:02.023972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:02.023972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.023972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:02.024326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.024326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:02.024326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:02.024326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:02.024326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.024326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:02.024326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:02.024326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:49:02.024326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:02.024326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:02.024326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.024326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:02.024729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:02.024729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:02.024729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.024729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:02.024729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:02.024729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:02.024729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:02.024729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:02.024729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:02.024729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:02.024729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:02.024729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:02.024729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:02.024729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:02.025222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:02.025222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:02.025222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.025222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:02.025415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.025415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:02.025415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:02.025415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:02.025415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:02.025415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:02.025415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.025415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:02.025415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:02.025415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.025415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:02.025415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:02.025415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.025415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:02.025415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:02.025415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:02.025877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:02.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:02.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:02.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:02.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:02.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:02.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:02.042340 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:02.042507 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:02.042637 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:02.042746 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:02.043761 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:02.043761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.043905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:02.043905 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:02.043905 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:02.043905 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:02.043905 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:02.043905 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:02.043905 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.043905 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:02.044534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.044534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:02.044534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:02.044534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:02.044534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.044534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:02.044534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:02.044534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:02.044534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.044534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:02.044534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:02.044534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:02.044534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:02.044534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:02.044534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:02.045173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:02.045173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:02.045173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:02.045173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:02.045173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:02.045173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:02.045173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:02.045173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.045173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:02.045511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.045511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:02.045511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:02.045511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:02.045511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.045511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:02.045511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:02.045511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:02.045511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:02.045511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.045511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:02.045731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:02.045731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:02.045731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.045731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:02.045731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:02.045731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:02.045731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:02.045731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:02.045731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:02.045731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:02.045731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:02.045731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:02.045731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:02.045731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:02.045945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:02.045945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:02.045945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.045945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.046267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:02.046808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:02.047493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:02.047493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:02.047493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.047493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:02.276434 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:02.276745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:02.276858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:02.276974 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:02.278110 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:02.278110 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:02.278110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.278274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:02.278274 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:02.278274 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:02.278274 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:02.278274 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:02.278274 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:02.278274 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.278274 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:02.279252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.279252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:02.279252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:02.279252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:02.279252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.279252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:02.279252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:02.279252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.279252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:02.279252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:02.279252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:02.279252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.279772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:02.279772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:02.279772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:02.279772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:02.279772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:02.279772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:02.279772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:02.279772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:02.279772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:02.279772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:02.279772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:02.279772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:02.279772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:02.280257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.280257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:02.280257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.280257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:02.280257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:02.280257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:02.280257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:02.280257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:02.280257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.280257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:02.280257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:02.280257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.280257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:02.280257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:02.280257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:02.280608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:02.280900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:02.280900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:02.280900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:02.280900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:02.280900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.280900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:02.290801 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:02.291094 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:02.291225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:02.291896 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:02.317769 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:02.317769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.317995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:02.317995 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:02.317995 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:02.317995 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:02.317995 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:02.317995 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:02.317995 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.317995 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:02.318750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.318750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:02.318750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:02.318750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:02.318750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.318750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:02.318750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:02.318750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.318750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:02.318750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:02.318750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:02.318750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.319039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:02.319039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:02.319039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:02.319039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:02.319039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:02.319039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:02.319039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:02.319039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:02.319039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:02.319039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:02.319039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:02.319039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:02.319039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:02.319243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.319243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:02.319481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.319481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:02.319481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:02.319481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:02.319481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:02.319481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:02.319481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.319481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:02.319481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:02.319481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.319481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:02.319481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:02.319481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.319481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:02.319481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:02.319481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:02.320090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:02.320737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:02.320737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:02.320737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:06.963719 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:06.964217 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:06.964217 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:06.964744 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:06.964904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:06.965010 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:06.965161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:06.965623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:06.965623 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:06.965623 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:06.965623 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:06.965623 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:06.965623 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:06.965623 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:06.965623 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:06.966222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:06.966222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:06.966222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:06.966222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:06.966222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:06.966222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:06.966222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:06.966222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:06.966222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:06.966222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:06.966222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:06.966222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:06.966752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:06.966752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:06.966752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:06.966752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:06.966752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:06.966752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:06.966752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:06.966752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:06.966752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:06.966752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:06.966752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:06.966752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:06.966752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:06.967520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:06.967520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:06.967520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:06.967520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:06.967520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:06.967520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:06.967520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:06.967520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:06.967520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:06.967520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:06.967520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:06.967520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:06.967520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:06.967520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:06.967520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:06.968462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:06.968884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:06.968884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:06.968884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:06.968884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:06.968884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:06.968884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:07.021431 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:07.021802 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:07.021881 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:07.021978 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:49:07.022288 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:07.022438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.022888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:07.022888 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:07.022888 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:07.022888 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:07.022888 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:07.022888 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:07.022888 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.022888 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:07.023387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.023387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:07.023387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:07.023387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:07.023387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.023387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:07.023387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:07.023387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:07.023387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.023387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:07.023387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:07.023387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:07.023387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:07.023387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:07.023387 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:07.023720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:07.023720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:07.023720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:07.023720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:07.023720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:07.023720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:07.023720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:07.023720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.023720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:07.024030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.024030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:07.024030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:07.024030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:07.024030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.024030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:07.024030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:07.024030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:49:07.024030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:07.024030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:07.024030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.024030 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:07.024379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:07.024379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:07.024379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.024379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:07.024379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:07.024379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:07.024379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:07.024379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:07.024379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:07.024379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:07.024379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:07.024379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:07.024379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:07.024379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:07.024921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:07.024921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:07.024921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.024921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:07.024921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.024921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:07.024921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:07.024921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:07.024921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:07.024921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:07.024921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.024921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:07.024921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:07.024921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:07.025334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:07.025664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:07.025664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:07.025664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:07.025664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:07.025664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:07.025664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:07.025664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:07.025664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:07.025664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:07.025664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:07.025664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.025664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:07.042659 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:07.042894 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:07.042970 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:07.043074 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:07.043369 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:07.043577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.043917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:07.043917 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:07.043917 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:07.043917 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:07.043917 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:07.043917 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:07.043917 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.043917 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:07.044392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.044392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:07.044392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:07.044392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:07.044392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.044392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:07.044392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:07.044392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:07.044392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.044392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:07.044392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:07.044392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:07.044392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:07.044392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:07.044392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:07.044691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:07.044691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:07.044691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:07.044691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:07.044691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:07.044691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:07.044691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:07.044691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.044691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:07.044960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.044960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:07.044960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:07.044960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:07.044960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.044960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:07.044960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:07.044960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:07.044960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:07.044960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.044960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:07.045160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:07.045160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:07.045160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.045160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:07.045160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:07.045160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:07.045160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:07.045160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:07.045160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:07.045160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:07.045160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:07.045160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:07.045160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:07.045160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:07.045377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:07.045377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:07.045377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.045377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.045655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:07.045971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:07.046258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:07.046258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:07.046258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.046258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:07.276861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:07.277216 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:07.277334 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:07.277427 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:07.277844 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:07.278032 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:07.278145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.278644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:07.278644 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:07.278644 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:07.278644 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:07.278644 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:07.278644 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:07.278644 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.278644 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:07.279393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.279393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:07.279393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:07.279393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:07.279393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.279393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:07.279393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:07.279393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.279393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:07.279393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:07.279393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:07.279393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.279889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:07.279889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:07.279889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:07.279889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:07.279889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:07.279889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:07.279889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:07.279889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:07.279889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:07.279889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:07.279889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:07.279889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:07.279889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:07.280200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.280200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:07.280200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.280200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:07.280200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:07.280200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:07.280200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:07.280200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:07.280200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.280200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:07.280200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:07.280200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.280200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:07.280200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:07.280200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:07.280658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:07.281152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:07.281152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:07.281152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:07.281152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:07.281152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.281152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:07.291200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:07.291409 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:07.291506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:07.292075 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:07.315858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:07.316020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.316558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:07.316558 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:07.316558 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:07.316558 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:07.316558 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:07.316558 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:07.316558 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.316558 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:07.317219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.317219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:07.317219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:07.317219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:07.317219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.317219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:07.317219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:07.317219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.317219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:07.317219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:07.317219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:07.317219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.317504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:07.317504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:07.317504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:07.317504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:07.317504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:07.317504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:07.317504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:07.317504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:07.317504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:07.317504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:07.317504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:07.317504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:07.317504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:07.317730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.317730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:07.317912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.317912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:07.317912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:07.317912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:07.317912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:07.317912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:07.317912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.317912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:07.317912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:07.317912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.317912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:07.317912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:07.317912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.317912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:07.317912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:07.317912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:07.318605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:07.319002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:07.319002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:07.319002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:11.964152 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:11.964609 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:11.964658 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:11.965372 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:11.965614 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:11.965753 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:11.965872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:11.966319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:11.966319 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:11.966319 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:11.966319 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:11.966319 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:11.966319 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:11.966319 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:11.966319 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:11.967062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:11.967062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:11.967062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:11.967062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:11.967062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:11.967062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:11.967062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:11.967062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:11.967062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:11.967062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:11.967062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:11.967062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:11.967464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:11.967464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:11.967464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:11.967464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:11.967464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:11.967464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:11.967464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:11.967464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:11.967464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:11.967464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:11.967464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:11.967464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:11.967464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:11.967902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:11.967902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:11.967902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:11.967902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:11.967902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:11.967902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:11.967902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:11.967902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:11.967902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:11.967902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:11.967902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:11.967902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:11.967902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:11.967902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:11.967902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:11.968367 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:11.968945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:11.968945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:11.968945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:11.968945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:11.968945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:11.968945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:12.021858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:12.022239 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:12.022328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:12.022431 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:49:12.023011 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:12.023304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.023892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:12.023892 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:12.023892 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:12.023892 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:12.023892 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:12.023892 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:12.023892 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.023892 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:12.024493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.024493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:12.024493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:12.024493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:12.024493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.024493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:12.024493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:12.024493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:12.024493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.024493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:12.024493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:12.024493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:12.024493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:12.024493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:12.024493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:12.024966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:12.024966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:12.024966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:12.024966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:12.024966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:12.024966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:12.024966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:12.024966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.024966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:12.025241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.025241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:12.025241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:12.025241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:12.025241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.025241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:12.025241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:12.025241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:49:12.025241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:12.025241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:12.025241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.025241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:12.025729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:12.025729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:12.025729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.025729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:12.025729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:12.025729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:12.025729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:12.025729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:12.025729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:12.025729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:12.025729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:12.025729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:12.025729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:12.025729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:12.026141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:12.026141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:12.026141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.026141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:12.026141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.026141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:12.026141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:12.026141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:12.026141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:12.026141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:12.026141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.026141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:12.026141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:12.026141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:12.026603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:12.027157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:12.027157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:12.027157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:12.027157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:12.027157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:12.027157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:12.027157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:12.027157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:12.027157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:12.027157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:12.027157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.027157 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:12.043019 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:12.043353 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:12.043447 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:12.043581 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:12.043994 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:12.044207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.044673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:12.044673 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:12.044673 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:12.044673 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:12.044673 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:12.044673 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:12.044673 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.044673 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:12.045258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.045258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:12.045258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:12.045258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:12.045258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.045258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:12.045258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:12.045258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:12.045258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.045258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:12.045258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:12.045258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:12.045258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:12.045258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:12.045258 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:12.045726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:12.045726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:12.045726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:12.045726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:12.045726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:12.045726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:12.045726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:12.045726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.045726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:12.046004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.046004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:12.046004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:12.046004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:12.046004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.046004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:12.046004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:12.046004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:12.046004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:12.046004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.046004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:12.046369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:12.046369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:12.046369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.046369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:12.046369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:12.046369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:12.046369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:12.046369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:12.046369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:12.046369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:12.046369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:12.046369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:12.046369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:12.046369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:12.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:12.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:12.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:12.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:12.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:12.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:12.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:12.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:12.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:12.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:12.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:12.047252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:12.047855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:12.047855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:12.047855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:12.047855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:12.047855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:12.047855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:12.047855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:12.047855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:12.047855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:12.047855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.047855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:12.277267 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:12.277546 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:12.277668 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:12.277816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:12.278826 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:12.279009 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:12.279134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.279524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:12.279524 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:12.279524 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:12.279524 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:12.279524 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:12.279524 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:12.279524 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.279524 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:12.280139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.280139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:12.280139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:12.280139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:12.280139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.280139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:12.280139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:12.280139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.280139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:12.280139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:12.280139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:12.280139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.280370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:12.280370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:12.280370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:12.280370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:12.280370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:12.280370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:12.280370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:12.280370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:12.280370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:12.280370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:12.280370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:12.280370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:12.280370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:12.280671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.280671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:12.280914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.280914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:12.280914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:12.280914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:12.280914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:12.280914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:12.280914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.280914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:12.280914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:12.280914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.280914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:12.280914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:12.280914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.280914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:12.280914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:12.280914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:12.281197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:12.281561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:12.281561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.281561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:12.291490 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:12.291671 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:12.291821 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:12.292338 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:12.317507 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:12.317668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.318061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:12.318061 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:12.318061 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:12.318061 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:12.318061 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:12.318061 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:12.318061 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.318061 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:12.318709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.318709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:12.318709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:12.318709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:12.318709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.318709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:12.318709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:12.318709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.318709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:12.318709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:12.318709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:12.318709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.318958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:12.318958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:12.318958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:12.318958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:12.318958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:12.318958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:12.318958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:12.318958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:12.318958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:12.318958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:12.318958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:12.318958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:12.318958 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:12.319172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.319172 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:12.319374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.319374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:12.319374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:12.319374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:12.319374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:12.319374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:12.319374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.319374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:12.319374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:12.319374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.319374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:12.319374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:12.319374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.319374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:12.319374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:12.319374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:12.319739 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:12.320094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:12.320094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:12.320094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:16.964662 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:16.965102 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:16.965102 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:16.965687 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:16.965901 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:16.965955 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:16.966043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:16.966455 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:16.966455 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:16.966455 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:16.966455 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:16.966455 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:16.966455 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:16.966455 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:16.966455 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:16.967290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:16.967290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:16.967290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:16.967290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:16.967290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:16.967290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:16.967290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:16.967290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:16.967290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:16.967290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:16.967290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:16.967290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:16.967648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:16.967648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:16.967648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:16.967648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:16.967648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:16.967648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:16.967648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:16.967648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:16.967648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:16.967648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:16.967648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:16.967648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:16.967648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:16.967928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:16.967928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:16.968020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:16.968020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:16.968020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:16.968020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:16.968020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:16.968020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:16.968020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:16.968020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:16.968020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:16.968020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:16.968020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:16.968020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:16.968020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:16.968020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:16.968020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:16.968020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:16.968318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:16.968764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:16.968764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:16.968764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:17.022266 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:17.022639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:17.022752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:17.022850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:49:17.023201 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:17.023383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.023822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:17.023822 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:17.023822 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:17.023822 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:17.023822 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:17.023822 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:17.023822 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.023822 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:17.024341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.024341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:17.024341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:17.024341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:17.024341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.024341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:17.024341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:17.024341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:17.024341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.024341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:17.024341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:17.024341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:17.024341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:17.024341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:17.024341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:17.024847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:17.024847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:17.024847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:17.024847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:17.024847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:17.024847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:17.024847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:17.024847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.024847 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:17.025122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.025122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:17.025122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:17.025122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:17.025122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.025122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:17.025122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:17.025122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:49:17.025122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:17.025122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:17.025122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.025122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:17.025443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:17.025443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:17.025443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.025443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:17.025443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:17.025443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:17.025443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:17.025443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:17.025443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:17.025443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:17.025443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:17.025443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:17.025443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:17.025443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:17.025813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:17.025813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:17.025813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.025813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:17.025813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.025813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:17.025813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:17.025813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:17.025813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:17.025813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:17.025813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.025813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:17.025813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:17.025813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:17.026190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:17.026815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:17.026815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:17.026815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:17.026815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:17.026815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:17.026815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:17.026815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:17.026815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:17.026815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:17.026815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:17.026815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.026815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:17.043389 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:17.043681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:17.043774 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:17.043874 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:17.044156 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:17.044346 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.044724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:17.044724 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:17.044724 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:17.044724 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:17.044724 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:17.044724 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:17.044724 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.044724 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:17.045200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.045200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:17.045200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:17.045200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:17.045200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.045200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:17.045200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:17.045200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:17.045200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.045200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:17.045200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:17.045200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:17.045200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:17.045200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:17.045200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:17.045625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:17.045625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:17.045625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:17.045625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:17.045625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:17.045625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:17.045625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:17.045625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.045625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:17.045855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.045855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:17.045855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:17.045855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:17.045855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.045855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:17.045855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:17.045855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:17.045855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:17.045855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.045855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:17.046173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:17.046173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:17.046173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.046173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:17.046173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:17.046173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:17.046173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:17.046173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:17.046173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:17.046173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:17.046173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:17.046173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:17.046173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:17.046173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:17.046534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:17.046534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:17.046534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.046534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:17.046534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.046534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:17.046534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:17.046534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:17.046534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:17.046534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:17.046534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.046534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:17.046534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:17.046534 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:17.046931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:17.047404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:17.047404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:17.047404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:17.047404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:17.047404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:17.047404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:17.047404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:17.047404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:17.047404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:17.047404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.047404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:17.277701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:17.278064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:17.278173 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:17.278321 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:17.278675 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:17.278857 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:17.278982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.279411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:17.279411 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:17.279411 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:17.279411 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:17.279411 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:17.279411 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:17.279411 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.279411 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:17.280046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.280046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:17.280046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:17.280046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:17.280046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.280046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:17.280046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:17.280046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.280046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:17.280046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:17.280046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:17.280046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.280463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:17.280463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:17.280463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:17.280463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:17.280463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:17.280463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:17.280463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:17.280463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:17.280463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:17.280463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:17.280463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:17.280463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:17.280463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:17.280853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.280853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:17.280853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.280853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:17.280853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:17.280853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:17.280853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:17.280853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:17.280853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.280853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:17.280853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:17.280853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.280853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:17.280853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:17.280853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:17.281292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:17.281857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:17.281857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:17.281857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:17.281857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:17.281857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.281857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:17.291866 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:17.292166 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:17.292286 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:17.292927 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:17.317822 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:17.317985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.318365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:17.318365 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:17.318365 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:17.318365 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:17.318365 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:17.318365 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:17.318365 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.318365 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:17.318972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.318972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:17.318972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:17.318972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:17.318972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.318972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:17.318972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:17.318972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.318972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:17.318972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:17.318972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:17.318972 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.319349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:17.319349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:17.319349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:17.319349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:17.319349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:17.319349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:17.319349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:17.319349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:17.319349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:17.319349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:17.319349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:17.319349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:17.319349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:17.319726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.319726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:17.319726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.319726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:17.319726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:17.319726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:17.319726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:17.319726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:17.319726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.319726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:17.319726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:17.319726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.319726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:17.319726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:17.319726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:17.320167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:17.320691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:17.320691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:17.320691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:17.320691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:17.320691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:17.320691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:21.965223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:21.965835 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:21.965835 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:21.966572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:21.966756 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:21.966876 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:21.966981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:21.967439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:21.967439 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:21.967439 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:21.967439 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:21.967439 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:21.967439 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:21.967439 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:21.967439 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:21.968292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:21.968292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:21.968292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:21.968292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:21.968292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:21.968292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:21.968292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:21.968292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:21.968292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:21.968292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:21.968292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:21.968292 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:21.968773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:21.968773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:21.968773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:21.968773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:21.968773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:21.968773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:21.968773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:21.968773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:21.968773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:21.968773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:21.968773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:21.968773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:21.968773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:21.969103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:21.969103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:21.969445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:21.969445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:21.969445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:21.969445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:21.969445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:21.969445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:21.969445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:21.969445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:21.969445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:21.969445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:21.969445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:21.969445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:21.969445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:21.969445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:21.969445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:21.969445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:21.969794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:21.970105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:21.970105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:21.970105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:22.022716 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:22.022975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:22.023103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:22.023252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:49:22.023776 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:22.023960 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.024430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:22.024430 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:22.024430 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:22.024430 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:22.024430 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:22.024430 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:22.024430 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.024430 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:22.025190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.025190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:22.025190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:22.025190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:22.025190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.025190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:22.025190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:22.025190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:22.025190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.025190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:22.025190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:22.025190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:22.025190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:22.025190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:22.025190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:22.025592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:22.025592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:22.025592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:22.025592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:22.025592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:22.025592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:22.025592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:22.025592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.025592 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:22.026146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.026146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:22.026146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:22.026146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:22.026146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.026146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:22.026146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:22.026146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:49:22.026146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:22.026146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:22.026146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.026146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:22.026605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:22.026605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:22.026605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.026605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:22.026605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:22.026605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:22.026605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:22.026605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:22.026605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:22.026605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:22.026605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:22.026605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:22.026605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:22.026605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:22.026995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:22.026995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:22.026995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.026995 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:22.027406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.027406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:22.027406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:22.027406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:22.027406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:22.027406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:22.027406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.027406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:22.027406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:22.027406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.027406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:22.027406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:22.027406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.027406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:22.027406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:22.027406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:22.028028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:22.028352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:22.028352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:22.028352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:22.028352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:22.028352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:22.028352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.028352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:22.043811 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:22.044099 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:22.044221 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:22.044375 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:22.044893 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:22.045087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.045729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:22.045729 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:22.045729 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:22.045729 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:22.045729 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:22.045729 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:22.045729 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.045729 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:22.046377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.046377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:22.046377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:22.046377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:22.046377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.046377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:22.046377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:22.046377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:22.046377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.046377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:22.046377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:22.046377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:22.046377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:22.046377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:22.046377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:22.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:22.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:22.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:22.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:22.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:22.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:22.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:22.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.046812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:22.047775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.047775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:22.047775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:22.047775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:22.047775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.047775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:22.047775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:22.047775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:22.047775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:22.047775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.047775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:22.048439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:22.048439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:22.048439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.048439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:22.048439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:22.048439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:22.048439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:22.048439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:22.048439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:22.048439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:22.048439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:22.048439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:22.048439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:22.048439 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:22.048846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:22.048846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:22.048846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.048846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:22.048846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.048846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:22.048846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:22.048846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:22.048846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:22.048846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:22.048846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.048846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:22.048846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:22.048846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:22.049070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:22.049362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:22.049362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:22.049362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:22.049362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:22.049362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:22.049362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:22.049362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:22.049362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:22.049362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:22.049362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.049362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:22.278117 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:22.278368 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:22.278465 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:22.278653 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:22.279103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:22.279315 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:22.279441 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.279946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:22.279946 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:22.279946 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:22.279946 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:22.279946 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:22.279946 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:22.279946 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.279946 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:22.280657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.280657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:22.280657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:22.280657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:22.280657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.280657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:22.280657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:22.280657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.280657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:22.280657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:22.280657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:22.280657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.280897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:22.280897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:22.280897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:22.280897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:22.280897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:22.280897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:22.280897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:22.280897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:22.280897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:22.280897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:22.280897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:22.280897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:22.280897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:22.281136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.281136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:22.281335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.281335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:22.281335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:22.281335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:22.281335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:22.281335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:22.281335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.281335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:22.281335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:22.281335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.281335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:22.281335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:22.281335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.281335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:22.281335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:22.281335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:22.281695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:22.282016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:22.282016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.282016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:22.292152 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:22.292335 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:22.292421 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:22.293022 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:22.294161 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=314,neutron:DELETE=10 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:49:22.321013 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:22.321274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.321954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:22.321954 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:22.321954 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:22.321954 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:22.321954 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:22.321954 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:22.321954 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.321954 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:22.322676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.322676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:22.322676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:22.322676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:22.322676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.322676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:22.322676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:22.322676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.322676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:22.322676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:22.322676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:22.322676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.322964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:22.322964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:22.322964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:22.322964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:22.322964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:22.322964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:22.322964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:22.322964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:22.322964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:22.322964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:22.322964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:22.322964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:22.322964 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:22.323206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.323206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:22.323333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.323333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:22.323333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:22.323333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:22.323333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:22.323333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:22.323333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.323333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:22.323333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:22.323333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.323333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:22.323333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:22.323333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.323333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:22.323333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:22.323333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:22.323780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:22.324119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:22.324119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:22.324119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:26.965613 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:26.966095 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:26.966095 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:26.966828 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:26.967030 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:26.967167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:26.967283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:26.967774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:26.967774 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:26.967774 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:26.967774 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:26.967774 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:26.967774 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:26.967774 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:26.967774 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:26.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:26.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:26.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:26.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:26.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:26.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:26.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:26.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:26.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:26.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:26.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:26.968469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:26.968878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:26.968878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:26.968878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:26.968878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:26.968878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:26.968878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:26.968878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:26.968878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:26.968878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:26.968878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:26.968878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:26.968878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:26.968878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:26.969126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:26.969126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:26.969934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:26.969934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:26.969934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:26.969934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:26.969934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:26.969934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:26.969934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:26.969934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:26.969934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:26.969934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:26.969934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:26.969934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:26.969934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:26.969934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:26.969934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:26.969934 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:26.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:26.971844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:26.971844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:26.971844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:27.023100 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:27.023599 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:27.023797 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:27.023984 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:49:27.024573 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:27.024771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.025236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:27.025236 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:27.025236 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:27.025236 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:27.025236 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:27.025236 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:27.025236 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.025236 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:27.025882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.025882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:27.025882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:27.025882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:27.025882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.025882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:27.025882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:27.025882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:27.025882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.025882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:27.025882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:27.025882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:27.025882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:27.025882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:27.025882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:27.026179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:27.026179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:27.026179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:27.026179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:27.026179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:27.026179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:27.026179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:27.026179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.026179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:27.026575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.026575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:27.026575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:27.026575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:27.026575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.026575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:27.026575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:27.026575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:49:27.026575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:27.026575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:27.026575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.026575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:27.027357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:27.027357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:27.027357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.027357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:27.027357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:27.027357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:27.027357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:27.027357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:27.027357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:27.027357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:27.027357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:27.027357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:27.027357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:27.027357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:27.027635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:27.027635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:27.027635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.027635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:27.027635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.027635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:27.027635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:27.027635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:27.027635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:27.027635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:27.027635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.027635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:27.027635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:27.027635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:27.027873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:27.028168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:27.028168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:27.028168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:27.028168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:27.028168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:27.028168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:27.028168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:27.028168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:27.028168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:27.028168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:27.028168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.028168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:27.044256 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:27.044660 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:27.044779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:27.044917 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:27.045877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:27.046075 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.046527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:27.046527 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:27.046527 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:27.046527 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:27.046527 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:27.046527 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:27.046527 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.046527 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:27.047178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.047178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:27.047178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:27.047178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:27.047178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.047178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:27.047178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:27.047178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:27.047178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.047178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:27.047178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:27.047178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:27.047178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:27.047178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:27.047178 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:27.047427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:27.047427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:27.047427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:27.047427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:27.047427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:27.047427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:27.047427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:27.047427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.047427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:27.047843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.047843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:27.047843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:27.047843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:27.047843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.047843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:27.047843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:27.047843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:27.047843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:27.047843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.047843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:27.048471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:27.048471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:27.048471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.048471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:27.048471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:27.048471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:27.048471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:27.048471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:27.048471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:27.048471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:27.048471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:27.048471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:27.048471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:27.048471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:27.500175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:27.500175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:27.500175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.500175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:27.500175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.500175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:27.500175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:27.500175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:27.500175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:27.500175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:27.500175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.500175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:27.500175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:27.500175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:27.501819 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:27.502603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:27.502603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:27.502603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:27.502603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:27.502603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:27.502603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:27.502603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:27.502603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:27.502603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:27.502603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.502603 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:27.502603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:27.502603 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:27.502965 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:27.502965 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:27.502965 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:27.502965 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:27.502965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.502965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:27.502965 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:27.502965 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:27.502965 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:27.503221 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:27.503221 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:27.503221 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.503221 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:27.503221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.503221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:27.503221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:27.503221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:27.503221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.503221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:27.503221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:27.503221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.503221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:27.503221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:27.503629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:27.503629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.503629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:27.503629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:27.503629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:27.503629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:27.503629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:27.503629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:27.503629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:27.503629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:27.503629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:27.503629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:27.503629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:27.503629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:27.503991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:27.503991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.503991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:27.503991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.503991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:27.503991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:27.503991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:27.503991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:27.503991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:27.503991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.503991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:27.503991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:27.503991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.503991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:27.503991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:27.503991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:27.504404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:27.505009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:27.505009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:27.505009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:27.505009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:27.505009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.505009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:27.505009 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:27.505009 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:27.505009 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:27.505009 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:27.505365 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:27.505365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.505365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:27.505365 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:27.505365 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:27.505365 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:27.505365 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:27.505365 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:27.505365 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.505365 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:27.505365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.505365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:27.505365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:27.505719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:27.505719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.505719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:27.505719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:27.505719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.505719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:27.505719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:27.505719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:27.505719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.505719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:27.505719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:27.505719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:27.505719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:27.505719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:27.506081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:27.506081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:27.506081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:27.506081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:27.506081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:27.506081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:27.506081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:27.506081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:27.506081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.506081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:27.506081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.506081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:27.506396 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:27.506906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:31.966031 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:31.966421 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:31.966549 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:31.967361 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:31.967567 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:31.967693 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:31.967798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:31.968289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:31.968289 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:31.968289 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:31.968289 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:31.968289 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:31.968289 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:31.968289 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:31.968289 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:31.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:31.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:31.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:31.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:31.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:31.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:31.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:31.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:31.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:31.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:31.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:31.969421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:31.969672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:31.969672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:31.969672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:31.969672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:31.969672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:31.969672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:31.969672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:31.969672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:31.969672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:31.969672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:31.969672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:31.969672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:31.969672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:31.969885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:31.969885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:31.970708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:31.970708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:31.970708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:31.970708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:31.970708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:31.970708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:31.970708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:31.970708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:31.970708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:31.970708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:31.970708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:31.970708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:31.970708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:31.970708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:31.970708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:31.970708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:31.970967 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:31.971246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:31.971246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:31.971246 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:32.023623 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:32.024031 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:32.024158 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:32.024255 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:49:32.024752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:32.024974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.025493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:32.025493 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:32.025493 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:32.025493 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:32.025493 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:32.025493 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:32.025493 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.025493 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:32.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:32.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:32.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:32.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:32.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:32.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:32.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:32.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:32.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:32.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:32.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:32.026197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:32.026475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:32.026475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:32.026475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:32.026475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:32.026475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:32.026475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:32.026475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:32.026475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.026475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:32.026912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.026912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:32.026912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:32.026912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:32.026912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.026912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:32.026912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:32.026912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:49:32.026912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:32.026912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:32.026912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.026912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:32.027115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:32.027115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:32.027115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.027115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:32.027115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:32.027115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:32.027115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:32.027115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:32.027115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:32.027115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:32.027115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:32.027115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:32.027115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:32.027115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:32.027310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:32.027310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:32.027310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.027310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:32.027800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.027800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:32.027800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:32.027800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:32.027800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:32.027800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:32.027800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.027800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:32.027800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:32.027800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.027800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:32.027800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:32.027800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.027800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:32.027800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:32.027800 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:32.028147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:32.028411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:32.028411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:32.028411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:32.028411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:32.028411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:32.028411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.028411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:32.044784 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:32.045078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:32.045232 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:32.045409 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:32.046109 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:32.046276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.046933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:32.046933 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:32.046933 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:32.046933 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:32.046933 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:32.046933 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:32.046933 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.046933 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:32.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:32.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:32.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:32.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:32.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:32.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:32.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:32.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:32.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:32.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:32.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:32.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:32.048016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:32.048016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:32.048016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:32.048016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:32.048016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:32.048016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:32.048016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:32.048016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.048016 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:32.048502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.048502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:32.048502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:32.048502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:32.048502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.048502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:32.048502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:32.048502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:32.048502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:32.048502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.048502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:32.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:32.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:32.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:32.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:32.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:32.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:32.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:32.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:32.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:32.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:32.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:32.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:32.048743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:32.048959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:32.048959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:32.048959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.048959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.049410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:32.049791 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:32.050202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:32.050202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:32.050202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.050202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:32.279086 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:32.279375 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:32.279521 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:32.279738 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:32.280268 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:32.280610 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:32.280827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.281453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:32.281453 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:32.281453 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:32.281453 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:32.281453 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:32.281453 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:32.281453 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.281453 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:32.282493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.282493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:32.282493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:32.282493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:32.282493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.282493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:32.282493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:32.282493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.282493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:32.282493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:32.282493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:32.282493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.282732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:32.282732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:32.282732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:32.282732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:32.282732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:32.282732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:32.282732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:32.282732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:32.282732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:32.282732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:32.282732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:32.282732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:32.282732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:32.282918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.282918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:32.283579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.283579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:32.283579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:32.283579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:32.283579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:32.283579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:32.283579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.283579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:32.283579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:32.283579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.283579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:32.283579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:32.283579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.283579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:32.283579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:32.283579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:32.283837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:32.284117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:32.284117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.284117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:32.292877 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:32.293138 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:32.293265 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:32.294226 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:32.327917 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:32.328124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.328706 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:32.328706 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:32.328706 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:32.328706 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:32.328706 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:32.328706 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:32.328706 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.328706 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:32.329707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.329707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:32.329707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:32.329707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:32.329707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.329707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:32.329707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:32.329707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.329707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:32.329707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:32.329707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:32.329707 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.329924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:32.329924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:32.329924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:32.329924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:32.329924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:32.329924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:32.329924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:32.329924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:32.329924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:32.329924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:32.329924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:32.329924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:32.329924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:32.330124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.330124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:32.330809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.330809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:32.330809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:32.330809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:32.330809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:32.330809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:32.330809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.330809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:32.330809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:32.330809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.330809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:32.330809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:32.330809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.330809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:32.330809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:32.330809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:32.331058 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:32.331332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:32.331332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:32.331332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:36.966515 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:36.967042 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:36.967042 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:36.967561 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:36.967731 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:36.967861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:36.967985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:36.968419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:36.968419 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:36.968419 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:36.968419 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:36.968419 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:36.968419 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:36.968419 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:36.968419 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:36.969259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:36.969259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:36.969259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:36.969259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:36.969259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:36.969259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:36.969259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:36.969259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:36.969259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:36.969259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:36.969259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:36.969259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:36.969458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:36.969458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:36.969458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:36.969458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:36.969458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:36.969458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:36.969458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:36.969458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:36.969458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:36.969458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:36.969458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:36.969458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:36.969458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:36.969696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:36.969696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:36.970161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:36.970161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:36.970161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:36.970161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:36.970161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:36.970161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:36.970161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:36.970161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:36.970161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:36.970161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:36.970161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:36.970161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:36.970161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:36.970161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:36.970161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:36.970161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:36.970412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:36.970722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:36.970722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:36.970722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:37.024127 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:37.024362 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:37.024462 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:37.024610 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:49:37.024932 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:37.025205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.025639 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:37.025639 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:37.025639 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:37.025639 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:37.025639 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:37.025639 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:37.025639 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.025639 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:37.026350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.026350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:37.026350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:37.026350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:37.026350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.026350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:37.026350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:37.026350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:37.026350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.026350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:37.026350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:37.026350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:37.026350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:37.026350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:37.026350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:37.026633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:37.026633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:37.026633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:37.026633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:37.026633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:37.026633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:37.026633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:37.026633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.026633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:37.027066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.027066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:37.027066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:37.027066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:37.027066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.027066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:37.027066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:37.027066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:49:37.027066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:37.027066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:37.027066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.027066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:37.027697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:37.027697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:37.027697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.027697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:37.027697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:37.027697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:37.027697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:37.027697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:37.027697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:37.027697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:37.027697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:37.027697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:37.027697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:37.027697 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:37.028200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:37.028200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:37.028200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.028200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:37.028200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.028200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:37.028200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:37.028200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:37.028200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:37.028200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:37.028200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.028200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:37.028200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:37.028200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:37.028435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:37.028802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:37.028802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:37.028802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:37.028802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:37.028802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:37.028802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:37.028802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:37.028802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:37.028802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:37.028802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:37.028802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.028802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:37.045178 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:37.045452 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:37.045593 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:37.045740 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:37.046087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:37.046274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.046725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:37.046725 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:37.046725 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:37.046725 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:37.046725 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:37.046725 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:37.046725 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.046725 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:37.047378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.047378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:37.047378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:37.047378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:37.047378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.047378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:37.047378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:37.047378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:37.047378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.047378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:37.047378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:37.047378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:37.047378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:37.047378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:37.047378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:37.047771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:37.047771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:37.047771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:37.047771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:37.047771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:37.047771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:37.047771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:37.047771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.047771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:37.048330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.048330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:37.048330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:37.048330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:37.048330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.048330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:37.048330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:37.048330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:37.048330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:37.048330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.048330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:37.048646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:37.048646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:37.048646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.048646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:37.048646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:37.048646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:37.048646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:37.048646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:37.048646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:37.048646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:37.048646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:37.048646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:37.048646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:37.048646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:37.048976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:37.048976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:37.048976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.048976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.049524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:37.049849 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:37.050129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:37.050129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:37.050129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.050129 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:37.279516 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:37.279816 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:37.279919 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:37.280072 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:37.281323 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:37.281323 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:37.281323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.281453 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:37.281453 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:37.281453 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:37.281453 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:37.281453 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:37.281453 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:37.281453 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.281453 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:37.282270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.282270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:37.282270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:37.282270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:37.282270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.282270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:37.282270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:37.282270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.282270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:37.282270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:37.282270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:37.282270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.282645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:37.282645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:37.282645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:37.282645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:37.282645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:37.282645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:37.282645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:37.282645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:37.282645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:37.282645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:37.282645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:37.282645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:37.282645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:37.282983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.282983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:37.283264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.283264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:37.283264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:37.283264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:37.283264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:37.283264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:37.283264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.283264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:37.283264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:37.283264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.283264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:37.283264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:37.283264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.283264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:37.283264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:37.283264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:37.283730 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:37.284264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:37.284264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.284264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:37.293226 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:37.293465 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:37.293609 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:37.294236 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:37.324278 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:37.324278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.324452 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:37.324452 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:37.324452 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:37.324452 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:37.324452 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:37.324452 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:37.324452 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.324452 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:37.325229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.325229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:37.325229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:37.325229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:37.325229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.325229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:37.325229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:37.325229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.325229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:37.325229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:37.325229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:37.325229 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.325780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:37.325780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:37.325780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:37.325780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:37.325780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:37.325780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:37.325780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:37.325780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:37.325780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:37.325780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:37.325780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:37.325780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:37.325780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:37.326109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.326109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:37.326109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.326109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:37.326109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:37.326109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:37.326109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:37.326109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:37.326109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.326109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:37.326109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:37.326109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.326109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:37.326109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:37.326109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:37.326801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:37.327305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:37.327305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:37.327305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:37.327305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:37.327305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:37.327305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:41.966988 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:41.967453 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:41.967453 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:41.968060 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:41.968260 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:41.968404 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:41.968590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:41.969053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:41.969053 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:41.969053 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:41.969053 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:41.969053 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:41.969053 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:41.969053 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:41.969053 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:41.969724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:41.969724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:41.969724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:41.969724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:41.969724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:41.969724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:41.969724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:41.969724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:41.969724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:41.969724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:41.969724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:41.969724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:41.969940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:41.969940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:41.969940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:41.969940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:41.969940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:41.969940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:41.969940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:41.969940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:41.969940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:41.969940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:41.969940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:41.969940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:41.969940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:41.970188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:41.970188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:41.970419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:41.970419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:41.970419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:41.970419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:41.970419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:41.970419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:41.970419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:41.970419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:41.970419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:41.970419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:41.970419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:41.970419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:41.970419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:41.970419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:41.970419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:41.970419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:41.970737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:41.971036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:41.971036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:41.971036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:42.024531 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:42.024900 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:42.024982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:42.025078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:49:42.025520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:42.025774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.026259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:42.026259 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:42.026259 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:42.026259 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:42.026259 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:42.026259 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:42.026259 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.026259 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:42.026859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.026859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:42.026859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:42.026859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:42.026859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.026859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:42.026859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:42.026859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:42.026859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.026859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:42.026859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:42.026859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:42.026859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:42.026859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:42.026859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:42.027126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:42.027126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:42.027126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:42.027126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:42.027126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:42.027126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:42.027126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:42.027126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.027126 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:42.027404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.027404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:42.027404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:42.027404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:42.027404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.027404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:42.027404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:42.027404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:49:42.027404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:42.027404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:42.027404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.027404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:42.027920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:42.027920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:42.027920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.027920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:42.027920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:42.027920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:42.027920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:42.027920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:42.027920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:42.027920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:42.027920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:42.027920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:42.027920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:42.027920 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:42.028371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:42.028371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:42.028371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.028371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:42.028371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.028371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:42.028371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:42.028371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:42.028371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:42.028371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:42.028371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.028371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:42.028371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:42.028371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:42.028651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:42.028937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:42.028937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:42.028937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:42.028937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:42.028937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:42.028937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:42.028937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:42.028937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:42.028937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:42.028937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:42.028937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.028937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:42.045581 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:42.045913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:42.046001 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:42.046097 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:42.046452 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:42.046720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.047166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:42.047166 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:42.047166 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:42.047166 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:42.047166 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:42.047166 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:42.047166 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.047166 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:42.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:42.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:42.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:42.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:42.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:42.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:42.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:42.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:42.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:42.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:42.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:42.047743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:42.047979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:42.047979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:42.047979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:42.047979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:42.047979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:42.047979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:42.047979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:42.047979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.047979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:42.048285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.048285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:42.048285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:42.048285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:42.048285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.048285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:42.048285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:42.048285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:42.048285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:42.048285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.048285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:42.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:42.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:42.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:42.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:42.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:42.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:42.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:42.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:42.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:42.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:42.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:42.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:42.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:42.048760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:42.048760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:42.048760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.048760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.049011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:42.049618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:42.050018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:42.050018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:42.050018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.050018 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:42.279982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:42.280351 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:42.280438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:42.280642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:42.281128 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:42.281371 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:42.281557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.282043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:42.282043 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:42.282043 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:42.282043 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:42.282043 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:42.282043 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:42.282043 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.282043 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:42.282740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.282740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:42.282740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:42.282740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:42.282740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.282740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:42.282740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:42.282740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.282740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:42.282740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:42.282740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:42.282740 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.283073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:42.283073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:42.283073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:42.283073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:42.283073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:42.283073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:42.283073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:42.283073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:42.283073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:42.283073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:42.283073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:42.283073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:42.283073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:42.283370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.283370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:42.283370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.283370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:42.283370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:42.283370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:42.283370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:42.283370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:42.283370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.283370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:42.283370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:42.283370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.283370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:42.283370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:42.283370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:42.283786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:42.284241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:42.284241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:42.284241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:42.284241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:42.284241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.284241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:42.293647 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:42.293904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:42.294003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:42.294553 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:42.319129 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:42.319413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.320074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:42.320074 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:42.320074 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:42.320074 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:42.320074 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:42.320074 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:42.320074 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.320074 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:42.320808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.320808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:42.320808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:42.320808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:42.320808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.320808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:42.320808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:42.320808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.320808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:42.320808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:42.320808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:42.320808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.321104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:42.321104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:42.321104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:42.321104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:42.321104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:42.321104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:42.321104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:42.321104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:42.321104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:42.321104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:42.321104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:42.321104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:42.321104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:42.321314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.321314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:42.321607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.321607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:42.321607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:42.321607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:42.321607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:42.321607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:42.321607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.321607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:42.321607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:42.321607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.321607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:42.321607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:42.321607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.321607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:42.321607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:42.321607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:42.322248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:42.323020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:42.323020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:42.323020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:46.967377 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:46.967968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:46.967968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:46.969499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:46.969499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:46.969499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:46.969499 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:46.969663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:46.969663 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:46.969663 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:46.969663 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:46.969663 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:46.969663 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:46.969663 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:46.969663 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:46.970458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:46.970458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:46.970458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:46.970458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:46.970458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:46.970458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:46.970458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:46.970458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:46.970458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:46.970458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:46.970458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:46.970458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:46.970836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:46.970836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:46.970836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:46.970836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:46.970836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:46.970836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:46.970836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:46.970836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:46.970836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:46.970836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:46.970836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:46.970836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:46.970836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:46.971130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:46.971130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:46.972256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:46.972256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:46.972256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:46.972256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:46.972256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:46.972256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:46.972256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:46.972256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:46.972256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:46.972256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:46.972256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:46.972256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:46.972256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:46.972256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:46.972256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:46.972256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:46.973095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:46.973527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:46.973527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:46.973527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:47.025043 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:47.025937 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:47.025937 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:47.025937 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:49:47.026748 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:47.026748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.026841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:47.026841 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:47.026841 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:47.026841 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:47.026841 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:47.026841 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:47.026841 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.026841 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:47.027379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.027379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:47.027379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:47.027379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:47.027379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.027379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:47.027379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:47.027379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:47.027379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.027379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:47.027379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:47.027379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:47.027379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:47.027379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:47.027379 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:47.027770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:47.027770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:47.027770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:47.027770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:47.027770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:47.027770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:47.027770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:47.027770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.027770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:47.029374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.029374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:47.029374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:47.029374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:47.029374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.029374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:47.029374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:47.029374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:49:47.029374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:47.029374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:47.029374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.029374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:47.029638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:47.029638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:47.029638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.029638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:47.029638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:47.029638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:47.029638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:47.029638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:47.029638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:47.029638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:47.029638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:47.029638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:47.029638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:47.029638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:47.029833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:47.029833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:47.029833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.029833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:47.029833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.029833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:47.029833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:47.029833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:47.029833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:47.029833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:47.029833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.029833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:47.029833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:47.029833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:47.030224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:47.030619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:47.030619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:47.030619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:47.030619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:47.030619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:47.030619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:47.030619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:47.030619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:47.030619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:47.030619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:47.030619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.030619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:47.046140 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:47.046895 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:47.046895 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:47.046895 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:47.047197 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:47.047356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.049907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:47.050239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:47.050239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:47.050239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:47.050239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:47.050239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:47.050239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:47.050239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:47.050239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:47.050239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:47.050239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:47.050239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:47.050239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:47.050239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.050239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:47.050239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.050465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:47.050465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:47.050465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:47.050465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.050465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:47.050465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:47.050465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:47.050465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:47.050465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.050465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:47.050465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:47.050465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:47.050465 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.050731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:47.050731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:47.050731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:47.050731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:47.050731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:47.050731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:47.050731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:47.050731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:47.050731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:47.050731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:47.050731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:47.050731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:47.050731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:47.050949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.050949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:47.050949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.050949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:47.050949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:47.050949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:47.050949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:47.050949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:47.050949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.050949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:47.050949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:47.050949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.050949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:47.050949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:47.050949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:47.051194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:47.051464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:47.051464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:47.051464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:47.051464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:47.051464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:47.051464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:47.051464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.051464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:47.280402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:47.281111 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:47.281111 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:47.281111 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:47.281429 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:47.281666 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:47.281793 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.282590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:47.282590 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:47.282590 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:47.282590 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:47.282590 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:47.282590 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:47.282590 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.282590 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:47.282983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.282983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:47.282983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:47.282983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:47.282983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.282983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:47.282983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:47.282983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.282983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:47.282983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:47.282983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:47.282983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.283203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:47.283203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:47.283203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:47.283203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:47.283203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:47.283203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:47.283203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:47.283203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:47.283203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:47.283203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:47.283203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:47.283203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:47.283203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:47.283474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.283474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:47.283653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.283653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:47.283653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:47.283653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:47.283653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:47.283653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:47.283653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.283653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:47.283653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:47.283653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.283653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:47.283653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:47.283653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.283653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:47.283653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:47.283653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:47.283955 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:47.284262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:47.284262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.284262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:47.293962 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:47.294166 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:47.294254 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:47.294924 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:47.328426 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:47.328682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.329302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:47.329302 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:47.329302 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:47.329302 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:47.329302 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:47.329302 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:47.329302 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.329302 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:47.330145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.035 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.330145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:47.330145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:47.330145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:47.330145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.330145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:47.330145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:47.330145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.330145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:47.330145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:47.330145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:47.330145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.330517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:47.330517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:47.330517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:47.330517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:47.330517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:47.330517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:47.330517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:47.330517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:47.330517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:47.330517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:47.330517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:47.330517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:47.330517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:47.330917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.330917 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:47.331259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.331259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:47.331259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:47.331259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:47.331259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:47.331259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:47.331259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.331259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:47.331259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:47.331259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.331259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:47.331259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:47.331259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.331259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:47.331259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:47.331259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:47.331631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:47.331945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:47.331945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:47.331945 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:51.967865 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:51.968415 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:51.968415 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:51.968964 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:51.969172 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:51.969309 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:51.969432 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:51.969943 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:51.969943 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:51.969943 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:51.969943 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:51.969943 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:51.969943 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:51.969943 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:51.969943 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:51.970657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:51.970657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:51.970657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:51.970657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:51.970657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:51.970657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:51.970657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:51.970657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:51.970657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:51.970657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:51.970657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:51.970657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:51.970882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:51.970882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:51.970882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:51.970882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:51.970882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:51.970882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:51.970882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:51.970882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:51.970882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:51.970882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:51.970882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:51.970882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:51.970882 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:51.971081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:51.971081 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:51.971388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:51.971388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:51.971388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:51.971388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:51.971388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:51.971388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:51.971388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:51.971388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:51.971388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:51.971388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:51.971388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:51.971388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:51.971388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:51.971388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:51.971388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:51.971388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:51.972008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:51.972607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:51.972607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:51.972607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:52.025612 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:52.025889 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:52.025974 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:52.026077 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:49:52.026435 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:52.026632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.027019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:52.027019 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:52.027019 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:52.027019 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:52.027019 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:52.027019 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:52.027019 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.027019 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:52.027530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.027530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:52.027530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:52.027530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:52.027530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.027530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:52.027530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:52.027530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:52.027530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.027530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:52.027530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:52.027530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:52.027530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:52.027530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:52.027530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:52.027804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:52.027804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:52.027804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:52.027804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:52.027804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:52.027804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:52.027804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:52.027804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.027804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:52.028103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.028103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:52.028103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:52.028103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:52.028103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.028103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:52.028103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:52.028103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:49:52.028103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:52.028103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:52.028103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.028103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:52.028590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:52.028590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:52.028590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.028590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:52.028590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:52.028590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:52.028590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:52.028590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:52.028590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:52.028590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:52.028590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:52.028590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:52.028590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:52.028590 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:52.029054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:52.029054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:52.029054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.029054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:52.029054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.029054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:52.029054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:52.029054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:52.029054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:52.029054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:52.029054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.029054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:52.029054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:52.029054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:52.029283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:52.029610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:52.029610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:52.029610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:52.029610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:52.029610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:52.029610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:52.029610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:52.029610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:52.029610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:52.029610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:52.029610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.029610 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:52.046493 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:52.046704 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:52.046788 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:52.046895 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:52.047233 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:52.047430 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:52.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:52.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:52.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:52.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:52.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:52.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:52.048350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.048350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:52.048350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:52.048350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:52.048350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.048350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:52.048350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:52.048350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:52.048350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.048350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:52.048350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:52.048350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:52.048350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:52.048350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:52.048350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:52.048629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:52.048629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:52.048629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:52.048629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:52.048629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:52.048629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:52.048629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:52.048629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.048629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:52.049015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.049015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:52.049015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:52.049015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:52.049015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.049015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:52.049015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:52.049015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:52.049015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:52.049015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.049015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:52.049626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:52.049626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:52.049626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.049626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:52.049626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:52.049626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:52.049626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:52.049626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:52.049626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:52.049626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:52.049626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:52.049626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:52.049626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:52.049626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:52.049974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:52.049974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:52.049974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.049974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:52.049974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.049974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:52.049974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:52.049974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:52.049974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:52.049974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:52.049974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.049974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:52.049974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:52.049974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:52.050204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:52.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:52.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:52.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:52.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:52.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:52.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:52.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:52.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:52.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:52.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.050501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:52.280890 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:52.281131 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:52.281258 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:52.281438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:52.281944 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:52.282151 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:52.282276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.282765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:52.282765 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:52.282765 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:52.282765 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:52.282765 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:52.282765 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:52.282765 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.282765 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:52.283398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.283398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:52.283398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:52.283398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:52.283398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.283398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:52.283398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:52.283398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.283398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:52.283398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:52.283398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:52.283398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.283686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:52.283686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:52.283686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:52.283686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:52.283686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:52.283686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:52.283686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:52.283686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:52.283686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:52.283686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:52.283686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:52.283686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:52.283686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:52.284031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.284031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:52.284137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.284137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:52.284137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:52.284137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:52.284137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:52.284137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:52.284137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.284137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:52.284137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:52.284137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.284137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:52.284137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:52.284137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.284137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:52.284137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:52.284137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:52.284600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:52.285102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:52.285102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.285102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:52.294298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:52.294536 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:52.294651 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:52.295267 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:52.321050 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:52.321194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.321635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:52.321635 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:52.321635 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:52.321635 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:52.321635 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:52.321635 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:52.321635 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.321635 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:52.322334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.322334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:52.322334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:52.322334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:52.322334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.322334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:52.322334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:52.322334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.322334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:52.322334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:52.322334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:52.322334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.322721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:52.322721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:52.322721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:52.322721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:52.322721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:52.322721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:52.322721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:52.322721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:52.322721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:52.322721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:52.322721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:52.322721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:52.322721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:52.323085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.323085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:52.323395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.323395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:52.323395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:52.323395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:52.323395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:52.323395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:52.323395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.323395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:52.323395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:52.323395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.323395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:52.323395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:52.323395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.323395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:52.323395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:52.323395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:52.323714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:52.324014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:52.324014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:52.324014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:56.968369 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:56.968882 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:56.968882 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:56.969366 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:56.969554 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:56.969702 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:56.969817 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:56.970227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:56.970227 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:56.970227 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:56.970227 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:56.970227 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:56.970227 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:56.970227 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:56.970227 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:56.970850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:56.970850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:56.970850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:56.970850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:56.970850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:56.970850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:56.970850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:56.970850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:56.970850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:56.970850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:56.970850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:56.970850 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:56.971074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:56.971074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:56.971074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:56.971074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:56.971074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:56.971074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:56.971074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:56.971074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:56.971074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:56.971074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:56.971074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:56.971074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:56.971074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:56.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:56.971283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:56.971527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:56.971527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:56.971527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:56.971527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:56.971527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:56.971527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:56.971527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:56.971527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:56.971527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:56.971527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:56.971527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:56.971527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:56.971527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:56.971527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:56.971527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:56.971527 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:56.971805 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:56.972115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:56.972115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:56.972115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:57.025995 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:57.026229 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:57.026325 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:57.026421 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:49:57.026774 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:57.026961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.027388 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:57.027388 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:57.027388 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:57.027388 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:57.027388 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:57.027388 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:57.027388 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.027388 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:57.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:57.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:57.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:57.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:57.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:57.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:57.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:57.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:57.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:57.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:57.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:57.027983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:57.028464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:57.028464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:57.028464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:57.028464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:57.028464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:57.028464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:57.028464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:57.028464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.028464 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:57.028638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.028638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:57.028638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:57.028638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:57.028638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.028638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:57.028638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:57.028638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:49:57.028638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:57.028638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:57.028638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.028638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:57.028826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:57.028826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:57.028826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.028826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:57.028826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:57.028826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:57.028826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:57.028826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:57.028826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:57.028826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:57.028826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:57.028826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:57.028826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:57.028826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:57.029024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:57.029024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:57.029024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.029024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:57.029357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.029357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:57.029357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:57.029357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:57.029357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:57.029357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:57.029357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.029357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:57.029357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:57.029357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.029357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:57.029357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:57.029357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.029357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:57.029357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:57.029357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:57.029996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:57.030623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:57.030623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:57.030623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:57.030623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:57.030623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:57.030623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.030623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:57.046819 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:57.047076 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:57.047197 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:57.047333 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:57.047713 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:57.047942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:57.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:57.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:57.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:57.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:57.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:57.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.048474 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:57.049112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.049112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:49:57.049112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:57.049112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:57.049112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.049112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:57.049112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:57.049112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:49:57.049112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.049112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:49:57.049112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:57.049112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:49:57.049112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:57.049112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:49:57.049112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:49:57.049374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:57.049374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:49:57.049374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:57.049374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:49:57.049374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:49:57.049374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:57.049374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:49:57.049374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.049374 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:49:57.049759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.049759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:57.049759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:57.049759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:57.049759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.049759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:57.049759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:57.049759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:57.049759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:57.049759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.049759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:57.049940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:57.049940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:57.049940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.049940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:57.049940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:57.049940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:57.049940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:57.049940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:57.049940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:57.049940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:57.049940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:57.049940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:57.049940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:57.049940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:57.050162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:57.050162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:57.050162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.050162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.050521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:57.050820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:57.051110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:57.051110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:57.051110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.051110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:57.281278 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:57.281525 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:57.281679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:57.281856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:57.282142 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:57.282328 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:57.282445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.282929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:57.282929 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:57.282929 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:57.282929 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:57.282929 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:57.282929 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:57.282929 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.282929 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:57.283669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.283669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:57.283669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:57.283669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:57.283669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.283669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:57.283669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:57.283669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.283669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:57.283669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:57.283669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:57.283669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.283982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:57.283982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:57.283982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:57.283982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:57.283982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:57.283982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:57.283982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:57.283982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:57.283982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:57.283982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:57.283982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:57.283982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:57.283982 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:57.284301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.284301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:57.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:57.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:57.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:57.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:57.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:57.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:57.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:57.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:57.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:57.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:57.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:57.284724 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:57.285023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:57.285409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:57.285409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.285409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:49:57.294606 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:49:57.294796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:49:57.294919 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:49:57.295497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:49:57.321025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:49:57.321275 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.323176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:49:57.323176 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:57.323176 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:49:57.323176 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:49:57.323176 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:57.323176 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:49:57.323176 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.323176 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:49:57.323176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.323176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:49:57.323176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:57.323176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:49:57.323176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.323176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:57.323176 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:57.323477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.323477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:57.323477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:57.323477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:49:57.323477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.323477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:49:57.323477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:57.323477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:49:57.323477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:57.323477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:49:57.323477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:49:57.323477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:57.323477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:49:57.323477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:57.323723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:49:57.323723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:49:57.323723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:57.323723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:49:57.323723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.323723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:49:57.323723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.323723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:49:57.323723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:49:57.323723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:49:57.323723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:49:57.323723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:49:57.323723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.323723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:49:57.323936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:49:57.324732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:49:57.324732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:49:57.324732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:49:57.324732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:49:57.324732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:49:57.324732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:49:57.324732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:49:57.324732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:49:57.324732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:01.969024 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:01.970477 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:01.970477 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:01.970679 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:01.970798 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:01.970921 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:01.971029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:01.972806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:01.972806 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:01.972806 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:01.972806 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:01.972806 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:01.972806 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:01.972806 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:01.972806 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:01.972806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:01.972806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:01.972806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:01.972806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:01.972806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:01.972806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:01.972806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:01.973393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:01.973393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:01.973393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:01.973393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:01.973393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:01.973393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:01.973393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:01.973393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:01.973393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:01.973393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:01.973393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:01.973393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:01.973393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:01.973393 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:01.973650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:01.973650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:01.973650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:01.973650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:01.973650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:01.973650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:01.973650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:01.973650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:01.973650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:01.973650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:01.973650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:01.973650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:01.973650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:01.973650 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:01.973877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:01.974265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:01.974265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:01.974265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:01.974265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:01.974265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:01.974265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:01.974265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:01.974265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:01.974265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:01.974265 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:02.026414 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:02.027122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:02.027122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:02.027122 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:50:02.027337 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:02.027447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.029726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:02.030122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:02.030122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:02.030122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:02.030122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:02.030122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:02.030122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:02.030122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:02.030122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:02.030122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:02.030122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:02.030122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:02.030122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:02.030122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.030122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:02.030122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.030354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:02.030354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:02.030354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:02.030354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.030354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:02.030354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:02.030354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:50:02.030354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:02.030354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:02.030354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.030354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:02.030354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:02.030354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:02.030589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.030589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:02.030589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:02.030589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:02.030589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:02.030589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:02.030589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:02.030589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:02.030589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:02.030589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:02.030589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:02.030589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:02.030589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:02.030589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:02.030788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.030788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:02.030788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.030788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:02.030788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:02.030788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:02.030788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:02.030788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:02.030788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.030788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:02.030788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:02.030788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.030788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:02.030788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:02.030788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:02.031093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:02.031382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:02.031382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:02.031382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:02.031382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:02.031382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:02.031382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:02.031382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:02.031382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.031382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:02.047263 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:02.047771 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:02.048000 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:02.048235 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:02.048711 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:02.048927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.051997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:02.052585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:02.052585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:02.052585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:02.052585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:02.052585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:02.052585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:02.052585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:02.052585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:02.052585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:02.052585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:02.052585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:02.052585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:02.052585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.052585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:02.052585 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.053067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:02.053067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:02.053067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:02.053067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.053067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:02.053067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:02.053067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:02.053067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:02.053067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.053067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:02.053067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:02.053067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:02.053067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.053371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:02.053371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:02.053371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:02.053371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:02.053371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:02.053371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:02.053371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:02.053371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:02.053371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:02.053371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:02.053371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:02.053371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:02.053371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:02.053667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.053667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:02.053667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.053667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:02.053667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:02.053667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:02.053667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:02.053667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:02.053667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.053667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:02.053667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:02.053667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.053667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:02.053667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:02.053667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:02.054065 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:02.054504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:02.054504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:02.054504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:02.054504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:02.054504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:02.054504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:02.054504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.054504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:02.281727 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:02.282420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:02.282420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:02.282420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:02.282761 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:02.282967 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:02.283100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.283568 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:02.283568 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:02.283568 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:02.283568 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:02.283568 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:02.283568 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:02.283568 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.283568 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:02.285427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.285427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:02.285427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:02.285427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:02.285427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.285427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:02.285427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:02.285427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.285427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:02.285427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:02.285427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:02.285427 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.286099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:02.286099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:02.286099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:02.286099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:02.286099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:02.286099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:02.286099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:02.286099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:02.286099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:02.286099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:02.286099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:02.286099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:02.286099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:02.286662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.286662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:02.286662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.286662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:02.286662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:02.286662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:02.286662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:02.286662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:02.286662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.286662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:02.286662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:02.286662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.286662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:02.286662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:02.286662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:02.286892 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:02.287169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:02.287169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:02.287169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:02.287169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:02.287169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.287169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:02.294859 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:02.295107 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:02.295302 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:02.295996 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:02.320714 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:02.320923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.321591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:02.321591 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:02.321591 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:02.321591 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:02.321591 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:02.321591 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:02.321591 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.321591 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:02.322256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.322256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:02.322256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:02.322256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:02.322256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.322256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:02.322256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:02.322256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.322256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:02.322256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:02.322256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:02.322256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.322505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:02.322505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:02.322505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:02.322505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:02.322505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:02.322505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:02.322505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:02.322505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:02.322505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:02.322505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:02.322505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:02.322505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:02.322505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:02.322723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.322723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:02.323483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.323483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:02.323483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:02.323483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:02.323483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:02.323483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:02.323483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.323483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:02.323483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:02.323483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.323483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:02.323483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:02.323483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.323483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:02.323483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:02.323483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:02.323968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:02.324343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:02.324343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:02.324343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:06.970171 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:06.970777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:06.970855 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:06.971777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:06.972027 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:06.972240 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:06.972365 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:06.972963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:06.972963 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:06.972963 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:06.972963 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:06.972963 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:06.972963 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:06.972963 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:06.972963 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:06.973664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:06.973664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:06.973664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:06.973664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:06.973664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:06.973664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:06.973664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:06.973664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:06.973664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:06.973664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:06.973664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:06.973664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:06.974029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:06.974029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:06.974029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:06.974029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:06.974029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:06.974029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:06.974029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:06.974029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:06.974029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:06.974029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:06.974029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:06.974029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:06.974029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:06.974399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:06.974399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:06.974399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:06.974399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:06.974399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:06.974399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:06.974399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:06.974399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:06.974399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:06.974399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:06.974399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:06.974399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:06.974399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:06.974399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:06.974399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:06.974865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:06.975383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:06.975383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:06.975383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:06.975383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:06.975383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:06.975383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:07.027018 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:07.027367 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:07.027504 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:07.027682 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:50:07.028633 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:07.028633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.028970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:07.028970 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:07.028970 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:07.028970 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:07.028970 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:07.028970 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:07.028970 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.028970 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:07.029635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.029635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:07.029635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:07.029635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:07.029635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.029635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:07.029635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:07.029635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:07.029635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.029635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:07.029635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:07.029635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:07.029635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:07.029635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:07.029635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:07.029990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:07.029990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:07.029990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:07.029990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:07.029990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:07.029990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:07.029990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:07.029990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.029990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:07.030373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.030373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:07.030373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:07.030373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:07.030373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.030373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:07.030373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:07.030373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:50:07.030373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:07.030373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:07.030373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.030373 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:07.030607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:07.030607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:07.030607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.030607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:07.030607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:07.030607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:07.030607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:07.030607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:07.030607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:07.030607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:07.030607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:07.030607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:07.030607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:07.030607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:07.030815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:07.030815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:07.030815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.030815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:07.031310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.031310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:07.031310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:07.031310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:07.031310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:07.031310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:07.031310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.031310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:07.031310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:07.031310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.031310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:07.031310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:07.031310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.031310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:07.031310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:07.031310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:07.031646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:07.032013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:07.032013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:07.032013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:07.032013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:07.032013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:07.032013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.032013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:07.047939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:07.048124 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:07.048225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:07.048332 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:07.048796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:07.048990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.049456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:07.049456 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:07.049456 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:07.049456 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:07.049456 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:07.049456 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:07.049456 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.049456 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:07.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:07.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:07.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:07.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:07.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:07.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:07.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:07.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:07.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:07.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:07.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:07.050055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:07.050459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:07.050459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:07.050459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:07.050459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:07.050459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:07.050459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:07.050459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:07.050459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.050459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:07.050771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.050771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:07.050771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:07.050771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:07.050771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.050771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:07.050771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:07.050771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:07.050771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:07.050771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.050771 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:07.051087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:07.051087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:07.051087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.051087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:07.051087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:07.051087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:07.051087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:07.051087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:07.051087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:07.051087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:07.051087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:07.051087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:07.051087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:07.051087 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:07.051533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:07.051533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:07.051533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.051533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:07.051533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.051533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:07.051533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:07.051533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:07.051533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:07.051533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:07.051533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.051533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:07.051533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:07.051533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:07.051916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:07.052409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:07.052409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:07.052409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:07.052409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:07.052409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:07.052409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:07.052409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:07.052409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:07.052409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:07.052409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.052409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:07.282459 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:07.282819 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:07.282934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:07.283087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:07.283595 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:07.283797 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:07.283931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.284456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:07.284456 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:07.284456 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:07.284456 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:07.284456 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:07.284456 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:07.284456 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.284456 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:07.285600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.285600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:07.285600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:07.285600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:07.285600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.285600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:07.285600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:07.285600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.285600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:07.285600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:07.285600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:07.285600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.285865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:07.285865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:07.285865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:07.285865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:07.285865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:07.285865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:07.285865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:07.285865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:07.285865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:07.285865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:07.285865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:07.285865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:07.285865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:07.286083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.286083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:07.286775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.286775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:07.286775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:07.286775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:07.286775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:07.286775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:07.286775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.286775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:07.286775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:07.286775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.286775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:07.286775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:07.286775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.286775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:07.286775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:07.286775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:07.287309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:07.287820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:07.287820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.287820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:07.295317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:07.295768 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:07.295888 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:07.296740 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:07.326462 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:07.326663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.327080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:07.327080 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:07.327080 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:07.327080 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:07.327080 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:07.327080 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:07.327080 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.327080 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:07.327683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.327683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:07.327683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:07.327683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:07.327683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.327683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:07.327683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:07.327683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.327683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:07.327683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:07.327683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:07.327683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.327948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:07.327948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:07.327948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:07.327948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:07.327948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:07.327948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:07.327948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:07.327948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:07.327948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:07.327948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:07.327948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:07.327948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:07.327948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:07.328151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.328151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:07.328414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.328414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:07.328414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:07.328414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:07.328414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:07.328414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:07.328414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.328414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:07.328414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:07.328414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.328414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:07.328414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:07.328414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.328414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:07.328414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:07.328414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:07.328727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:07.329117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:07.329117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:07.329117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:11.970987 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:11.971375 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:11.971607 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:11.972262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:11.972520 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:11.972734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:11.972909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:11.973422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:11.973422 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:11.973422 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:11.973422 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:11.973422 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:11.973422 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:11.973422 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:11.973422 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:11.974334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:11.974334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:11.974334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:11.974334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:11.974334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:11.974334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:11.974334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:11.974334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:11.974334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:11.974334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:11.974334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:11.974334 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:11.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:11.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:11.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:11.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:11.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:11.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:11.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:11.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:11.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:11.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:11.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:11.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:11.974623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:11.974848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:11.974848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:11.975214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:11.975214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:11.975214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:11.975214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:11.975214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:11.975214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:11.975214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:11.975214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:11.975214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:11.975214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:11.975214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:11.975214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:11.975214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:11.975214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:11.975214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:11.975214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:11.975475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:11.975867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:11.975867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:11.975867 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:12.027517 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:12.027839 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:12.027950 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:12.028067 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:50:12.028390 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:12.028619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.029080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:12.029080 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:12.029080 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:12.029080 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:12.029080 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:12.029080 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:12.029080 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.029080 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:12.029666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.029666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:12.029666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:12.029666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:12.029666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.029666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:12.029666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:12.029666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:12.029666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.029666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:12.029666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:12.029666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:12.029666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:12.029666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:12.029666 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:12.029948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:12.029948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:12.029948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:12.029948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:12.029948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:12.029948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:12.029948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:12.029948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.029948 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:12.030262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.030262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:12.030262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:12.030262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:12.030262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.030262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:12.030262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:12.030262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:50:12.030262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:12.030262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:12.030262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.030262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:12.030521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:12.030521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:12.030521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.030521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:12.030521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:12.030521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:12.030521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:12.030521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:12.030521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:12.030521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:12.030521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:12.030521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:12.030521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:12.030521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:12.030778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:12.030778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:12.030778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.030778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:12.030966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.030966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:12.030966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:12.030966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:12.030966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:12.030966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:12.030966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.030966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:12.030966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:12.030966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.030966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:12.030966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:12.030966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.030966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:12.030966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:12.030966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:12.031235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:12.031588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:12.031588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:12.031588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:12.031588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:12.031588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:12.031588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.031588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:12.048314 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:12.048644 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:12.048771 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:12.048884 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:12.049208 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:12.049421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:12.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:12.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:12.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:12.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:12.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:12.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.049929 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:12.050509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.050509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:12.050509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:12.050509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:12.050509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.050509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:12.050509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:12.050509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:12.050509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.050509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:12.050509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:12.050509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:12.050509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:12.050509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:12.050509 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:12.050785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:12.050785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:12.050785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:12.050785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:12.050785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:12.050785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:12.050785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:12.050785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.050785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:12.051138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.051138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:12.051138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:12.051138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:12.051138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.051138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:12.051138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:12.051138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:12.051138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:12.051138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.051138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:12.051327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:12.051327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:12.051327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.051327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:12.051327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:12.051327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:12.051327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:12.051327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:12.051327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:12.051327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:12.051327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:12.051327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:12.051327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:12.051327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:12.051579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:12.051579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:12.051579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.051579 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.052026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:12.052304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:12.052656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:12.052656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:12.052656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.052656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:12.282842 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:12.283326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:12.283477 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:12.283695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:12.283985 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:12.284193 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:12.284356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.284824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:12.284824 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:12.284824 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:12.284824 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:12.284824 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:12.284824 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:12.284824 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.284824 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:12.285553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.285553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:12.285553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:12.285553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:12.285553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.285553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:12.285553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:12.285553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.285553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:12.285553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:12.285553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:12.285553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.285919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:12.285919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:12.285919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:12.285919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:12.285919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:12.285919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:12.285919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:12.285919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:12.285919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:12.285919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:12.285919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:12.285919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:12.285919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:12.286267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.286267 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:12.286622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.286622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:12.286622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:12.286622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:12.286622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:12.286622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:12.286622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.286622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:12.286622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:12.286622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.286622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:12.286622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:12.286622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.286622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:12.286622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:12.286622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:12.287090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:12.287625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:12.287625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.287625 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:12.295669 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:12.295885 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:12.296012 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:12.296623 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:12.323870 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:12.324037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.324463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:12.324463 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:12.324463 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:12.324463 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:12.324463 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:12.324463 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:12.324463 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.324463 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:12.325135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.325135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:12.325135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:12.325135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:12.325135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.325135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:12.325135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:12.325135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.325135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:12.325135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:12.325135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:12.325135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.325421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:12.325421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:12.325421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:12.325421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:12.325421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:12.325421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:12.325421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:12.325421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:12.325421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:12.325421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:12.325421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:12.325421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:12.325421 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:12.325682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.325682 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:12.325871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.325871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:12.325871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:12.325871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:12.325871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:12.325871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:12.325871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.325871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:12.325871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:12.325871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.325871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:12.325871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:12.325871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.325871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:12.325871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:12.325871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:12.326336 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:12.326874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:12.326874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:12.326874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:16.971535 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:16.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:16.972111 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:16.972799 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:16.972998 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:16.973152 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:16.973286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:16.973870 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:16.973870 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:16.973870 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:16.973870 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:16.973870 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:16.973870 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:16.973870 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:16.973870 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:16.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:16.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:16.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:16.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:16.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:16.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:16.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:16.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:16.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:16.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:16.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:16.974890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:16.975092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:16.975092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:16.975092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:16.975092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:16.975092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:16.975092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:16.975092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:16.975092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:16.975092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:16.975092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:16.975092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:16.975092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:16.975092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:16.975347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:16.975347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:16.976148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:16.976148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:16.976148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:16.976148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:16.976148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:16.976148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:16.976148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:16.976148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:16.976148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:16.976148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:16.976148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:16.976148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:16.976148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:16.976148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:16.976148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:16.976148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:16.976394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:16.976702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:16.976702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:16.976702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:17.027996 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:17.028325 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:17.028475 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:17.028619 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:50:17.028985 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:17.029192 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.029778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:17.029778 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:17.029778 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:17.029778 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:17.029778 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:17.029778 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:17.029778 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.029778 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:17.030581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.030581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:17.030581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:17.030581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:17.030581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.030581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:17.030581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:17.030581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:17.030581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.030581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:17.030581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:17.030581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:17.030581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:17.030581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:17.030581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:17.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:17.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:17.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:17.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:17.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:17.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:17.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:17.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:17.031481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.031481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:17.031481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:17.031481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:17.031481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.031481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:17.031481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:17.031481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:50:17.031481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:17.031481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:17.031481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.031481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:17.031715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:17.031715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:17.031715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.031715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:17.031715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:17.031715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:17.031715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:17.031715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:17.031715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:17.031715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:17.031715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:17.031715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:17.031715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:17.031715 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:17.032011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:17.032011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:17.032011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.032011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:17.032516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.032516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:17.032516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:17.032516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:17.032516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:17.032516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:17.032516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.032516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:17.032516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:17.032516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.032516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:17.032516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:17.032516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.032516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:17.032516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:17.032516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:17.032838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:17.033163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:17.033163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:17.033163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:17.033163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:17.033163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:17.033163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.033163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:17.048711 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:17.048956 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:17.049063 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:17.049198 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:17.049550 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:17.049815 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.050287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:17.050287 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:17.050287 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:17.050287 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:17.050287 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:17.050287 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:17.050287 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.050287 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:17.050951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.050951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:17.050951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:17.050951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:17.050951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.050951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:17.050951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:17.050951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:17.050951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.050951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:17.050951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:17.050951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:17.050951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:17.050951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:17.050951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:17.051223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:17.051223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:17.051223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:17.051223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:17.051223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:17.051223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:17.051223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:17.051223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.051223 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:17.051758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.051758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:17.051758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:17.051758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:17.051758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.051758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:17.051758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:17.051758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:17.051758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:17.051758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.051758 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:17.051969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:17.051969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:17.051969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.051969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:17.051969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:17.051969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:17.051969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:17.051969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:17.051969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:17.051969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:17.051969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:17.051969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:17.051969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:17.051969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:17.052242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:17.052242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:17.052242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.052242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.052875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:17.053215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:17.053611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:17.053611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:17.053611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.053611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:17.283274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:17.283791 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:17.283939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:17.284139 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:17.284582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:17.284850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:17.285039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.285670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:17.285670 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:17.285670 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:17.285670 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:17.285670 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:17.285670 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:17.285670 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.285670 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:17.286642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.286642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:17.286642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:17.286642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:17.286642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.286642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:17.286642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:17.286642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.286642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:17.286642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:17.286642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:17.286642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.287028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:17.287028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:17.287028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:17.287028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:17.287028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:17.287028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:17.287028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:17.287028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:17.287028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:17.287028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:17.287028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:17.287028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:17.287028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:17.287276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.287276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:17.287630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.287630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:17.287630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:17.287630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:17.287630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:17.287630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:17.287630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.287630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:17.287630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:17.287630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.287630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:17.287630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:17.287630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.287630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:17.287630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:17.287630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:17.288070 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:17.288349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:17.288349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.288349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:17.296062 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:17.296326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:17.296462 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:17.297361 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:17.335560 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:17.335752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.336174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:17.336174 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:17.336174 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:17.336174 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:17.336174 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:17.336174 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:17.336174 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.336174 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:17.336826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.040 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.336826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:17.336826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:17.336826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:17.336826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.336826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:17.336826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:17.336826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.336826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:17.336826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:17.336826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:17.336826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.337061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:17.337061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:17.337061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:17.337061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:17.337061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:17.337061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:17.337061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:17.337061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:17.337061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:17.337061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:17.337061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:17.337061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:17.337061 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:17.337256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.337256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:17.337586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.337586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:17.337586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:17.337586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:17.337586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:17.337586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:17.337586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.337586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:17.337586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:17.337586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.337586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:17.337586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:17.337586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.337586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:17.337586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:17.337586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:17.338201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:17.338693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:17.338693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:17.338693 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:21.972007 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:21.972681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:21.972681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:21.973495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:21.973719 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:21.973841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:21.973956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:21.974413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:21.974413 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:21.974413 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:21.974413 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:21.974413 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:21.974413 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:21.974413 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:21.974413 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:21.975291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:21.975291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:21.975291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:21.975291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:21.975291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:21.975291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:21.975291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:21.975291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:21.975291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:21.975291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:21.975291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:21.975291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:21.975757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:21.975757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:21.975757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:21.975757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:21.975757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:21.975757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:21.975757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:21.975757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:21.975757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:21.975757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:21.975757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:21.975757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:21.975757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:21.976349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:21.976349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:21.976457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:21.976457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:21.976457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:21.976457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:21.976457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:21.976457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:21.976457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:21.976457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:21.976457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:21.976457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:21.976457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:21.976457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:21.976457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:21.976457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:21.976457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:21.976457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:21.976807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:21.977140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:21.977140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:21.977140 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:22.028440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:22.028861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:22.029031 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:22.029193 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:50:22.029717 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:22.029885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.030327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:22.030327 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:22.030327 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:22.030327 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:22.030327 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:22.030327 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:22.030327 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.030327 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:22.031056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.031056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:22.031056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:22.031056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:22.031056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.031056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:22.031056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:22.031056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:22.031056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.031056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:22.031056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:22.031056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:22.031056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:22.031056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:22.031056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:22.031525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:22.031525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:22.031525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:22.031525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:22.031525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:22.031525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:22.031525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:22.031525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.031525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:22.031984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.031984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:22.031984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:22.031984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:22.031984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.031984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:22.031984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:22.031984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:50:22.031984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:22.031984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:22.031984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.031984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:22.032340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:22.032340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:22.032340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.032340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:22.032340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:22.032340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:22.032340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:22.032340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:22.032340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:22.032340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:22.032340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:22.032340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:22.032340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:22.032340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:22.032781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:22.032781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:22.032781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.032781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:22.033074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.033074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:22.033074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:22.033074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:22.033074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:22.033074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:22.033074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.033074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:22.033074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:22.033074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.033074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:22.033074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:22.033074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.033074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:22.033074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:22.033074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:22.033429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:22.033804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:22.033804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:22.033804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:22.033804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:22.033804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:22.033804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.033804 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:22.049087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:22.049347 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:22.049434 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:22.049547 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:22.049994 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:22.050214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.050664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:22.050664 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:22.050664 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:22.050664 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:22.050664 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:22.050664 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:22.050664 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.050664 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:22.051186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.051186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:22.051186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:22.051186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:22.051186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.051186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:22.051186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:22.051186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:22.051186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.051186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:22.051186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:22.051186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:22.051186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:22.051186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:22.051186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:22.051580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:22.051580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:22.051580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:22.051580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:22.051580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:22.051580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:22.051580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:22.051580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.051580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:22.051854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.051854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:22.051854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:22.051854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:22.051854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.051854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:22.051854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:22.051854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:22.051854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:22.051854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.051854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:22.052050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:22.052050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:22.052050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.052050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:22.052050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:22.052050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:22.052050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:22.052050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:22.052050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:22.052050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:22.052050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:22.052050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:22.052050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:22.052050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:22.052274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:22.052274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:22.052274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.052274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.052637 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:22.052918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:22.053235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:22.053235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:22.053235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.053235 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:22.283733 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:22.284012 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:22.284126 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:22.284266 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:22.284839 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:22.285124 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:22.285321 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.285933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:22.285933 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:22.285933 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:22.285933 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:22.285933 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:22.285933 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:22.285933 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.285933 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:22.286763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.286763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:22.286763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:22.286763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:22.286763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.286763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:22.286763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:22.286763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.286763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:22.286763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:22.286763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:22.286763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.287004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:22.287004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:22.287004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:22.287004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:22.287004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:22.287004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:22.287004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:22.287004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:22.287004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:22.287004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:22.287004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:22.287004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:22.287004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:22.287208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.287208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:22.287687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.287687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:22.287687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:22.287687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:22.287687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:22.287687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:22.287687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.287687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:22.287687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:22.287687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.287687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:22.287687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:22.287687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.287687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:22.287687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:22.287687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:22.287959 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:22.288268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:22.288268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.288268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:22.296317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:22.296552 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:22.296671 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:22.297250 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:22.302701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=242 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:50:22.326154 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:22.326425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.327101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:22.327101 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:22.327101 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:22.327101 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:22.327101 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:22.327101 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:22.327101 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.327101 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:22.327872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.327872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:22.327872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:22.327872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:22.327872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.327872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:22.327872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:22.327872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.327872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:22.327872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:22.327872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:22.327872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.328151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:22.328151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:22.328151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:22.328151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:22.328151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:22.328151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:22.328151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:22.328151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:22.328151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:22.328151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:22.328151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:22.328151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:22.328151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:22.328371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.328371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:22.328826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.328826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:22.328826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:22.328826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:22.328826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:22.328826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:22.328826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.328826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:22.328826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:22.328826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.328826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:22.328826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:22.328826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.328826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:22.328826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:22.328826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:22.329110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:22.329429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:22.329429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:22.329429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:26.972478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:26.972982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:26.972982 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:26.973261 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:26.973401 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:26.973518 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:26.973664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:26.974053 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:26.974053 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:26.974053 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:26.974053 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:26.974053 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:26.974053 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:26.974053 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:26.974053 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:26.974662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:26.974662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:26.974662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:26.974662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:26.974662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:26.974662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:26.974662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:26.974662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:26.974662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:26.974662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:26.974662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:26.974662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:26.974899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:26.974899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:26.974899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:26.974899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:26.974899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:26.974899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:26.974899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:26.974899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:26.974899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:26.974899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:26.974899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:26.974899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:26.974899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:26.975202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:26.975202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:26.975345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:26.975345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:26.975345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:26.975345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:26.975345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:26.975345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:26.975345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:26.975345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:26.975345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:26.975345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:26.975345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:26.975345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:26.975345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:26.975345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:26.975345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:26.975345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:26.975966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:26.976767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:26.976767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:26.976767 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:27.028838 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:27.029060 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:27.029170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:27.029252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:50:27.029578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:27.029737 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.030145 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:27.030145 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:27.030145 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:27.030145 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:27.030145 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:27.030145 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:27.030145 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.030145 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:27.030738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.030738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:27.030738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:27.030738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:27.030738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.030738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:27.030738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:27.030738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:27.030738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.030738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:27.030738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:27.030738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:27.030738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:27.030738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:27.030738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:27.031027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:27.031027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:27.031027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:27.031027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:27.031027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:27.031027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:27.031027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:27.031027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.031027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:27.031299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.031299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:27.031299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:27.031299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:27.031299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.031299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:27.031299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:27.031299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:50:27.031299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:27.031299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:27.031299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.031299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:27.031566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:27.031566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:27.031566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.031566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:27.031566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:27.031566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:27.031566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:27.031566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:27.031566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:27.031566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:27.031566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:27.031566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:27.031566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:27.031566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:27.031801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:27.031801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:27.031801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.031801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:27.032237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.032237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:27.032237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:27.032237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:27.032237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:27.032237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:27.032237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.032237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:27.032237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:27.032237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.032237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:27.032237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:27.032237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.032237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:27.032237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:27.032237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:27.032833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:27.033470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:27.033470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:27.033470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:27.033470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:27.033470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:27.033470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.033470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:27.049497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:27.049784 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:27.049928 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:27.050084 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:27.050463 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:27.050735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.051264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:27.051264 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:27.051264 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:27.051264 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:27.051264 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:27.051264 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:27.051264 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.051264 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:27.052103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.052103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:27.052103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:27.052103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:27.052103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.052103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:27.052103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:27.052103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:27.052103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.052103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:27.052103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:27.052103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:27.052103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:27.052103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:27.052103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:27.052552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:27.052552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:27.052552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:27.052552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:27.052552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:27.052552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:27.052552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:27.052552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.052552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:27.053051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.053051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:27.053051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:27.053051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:27.053051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.053051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:27.053051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:27.053051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:27.053051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:27.053051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.053051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:27.053399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:27.053399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:27.053399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.053399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:27.053399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:27.053399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:27.053399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:27.053399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:27.053399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:27.053399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:27.053399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:27.053399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:27.053399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:27.053399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:27.053812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:27.053812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:27.053812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.053812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.054299 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:27.054807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:27.055331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:27.055331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:27.055331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.055331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:27.284126 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:27.284339 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:27.284427 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:27.284610 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:27.284899 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:27.285125 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:27.285218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.285698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:27.285698 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:27.285698 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:27.285698 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:27.285698 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:27.285698 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:27.285698 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.285698 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:27.286226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.286226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:27.286226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:27.286226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:27.286226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.286226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:27.286226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:27.286226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.286226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:27.286226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:27.286226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:27.286226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.286466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:27.286466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:27.286466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:27.286466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:27.286466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:27.286466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:27.286466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:27.286466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:27.286466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:27.286466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:27.286466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:27.286466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:27.286466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:27.286750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.286750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:27.286930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.286930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:27.286930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:27.286930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:27.286930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:27.286930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:27.286930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.286930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:27.286930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:27.286930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.286930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:27.286930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:27.286930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.286930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:27.286930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:27.286930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:27.287613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:27.288263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:27.288263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.288263 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:27.296801 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:27.297164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:27.297264 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:27.298096 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:27.330627 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:27.330825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.331318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:27.331318 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:27.331318 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:27.331318 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:27.331318 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:27.331318 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:27.331318 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.331318 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:27.332032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.332032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:27.332032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:27.332032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:27.332032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.332032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:27.332032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:27.332032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.332032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:27.332032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:27.332032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:27.332032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.332287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:27.332287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:27.332287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:27.332287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:27.332287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:27.332287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:27.332287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:27.332287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:27.332287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:27.332287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:27.332287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:27.332287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:27.332287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:27.332559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.332559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:27.332855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.332855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:27.332855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:27.332855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:27.332855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:27.332855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:27.332855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.332855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:27.332855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:27.332855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.332855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:27.332855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:27.332855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.332855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:27.332855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:27.332855 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:27.333146 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:27.333449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:27.333449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:27.333449 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:31.972810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:31.973428 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:31.973428 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:31.973754 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:31.973907 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:31.974026 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:31.974130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:31.974529 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:31.974529 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:31.974529 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:31.974529 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:31.974529 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:31.974529 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:31.974529 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:31.974529 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:31.975169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:31.975169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:31.975169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:31.975169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:31.975169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:31.975169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:31.975169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:31.975169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:31.975169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:31.975169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:31.975169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:31.975169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:31.975531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:31.975531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:31.975531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:31.975531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:31.975531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:31.975531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:31.975531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:31.975531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:31.975531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:31.975531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:31.975531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:31.975531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:31.975531 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:31.975755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:31.975755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:31.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:31.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:31.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:31.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:31.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:31.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:31.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:31.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:31.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:31.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:31.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:31.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:31.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:31.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:31.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:31.976040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:31.976722 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:31.977036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:31.977036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:31.977036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:32.029246 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:32.029503 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:32.029572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:32.029676 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:50:32.029978 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:32.030106 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:32.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:32.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:32.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:32.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:32.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:32.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.030864 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:32.031200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.031200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:32.031200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:32.031200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:32.031200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.031200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:32.031200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:32.031200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:32.031200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.031200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:32.031200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:32.031200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:32.031200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:32.031200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:32.031200 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:32.031615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:32.031615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:32.031615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:32.031615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:32.031615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:32.031615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:32.031615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:32.031615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.031615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:32.031828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.031828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:32.031828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:32.031828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:32.031828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.031828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:32.031828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:32.031828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:50:32.031828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:32.031828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:32.031828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.031828 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:32.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:32.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:32.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:32.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:32.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:32.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:32.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:32.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:32.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:32.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:32.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:32.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:32.032026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:32.032270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:32.032270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:32.032270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.032270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:32.032841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.032841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:32.032841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:32.032841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:32.032841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:32.032841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:32.032841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.032841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:32.032841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:32.032841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.032841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:32.032841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:32.032841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.032841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:32.032841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:32.032841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:32.034155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:32.035136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:32.035136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:32.035136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:32.035136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:32.035136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:32.035136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.035136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:32.049842 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:32.050065 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:32.050167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:32.050271 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:32.050551 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:32.050749 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.051117 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:32.051117 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:32.051117 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:32.051117 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:32.051117 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:32.051117 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:32.051117 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.051117 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:32.051683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.051683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:32.051683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:32.051683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:32.051683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.051683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:32.051683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:32.051683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:32.051683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.051683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:32.051683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:32.051683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:32.051683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:32.051683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:32.051683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:32.051946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:32.051946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:32.051946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:32.051946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:32.051946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:32.051946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:32.051946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:32.051946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.051946 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:32.052278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.052278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:32.052278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:32.052278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:32.052278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.052278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:32.052278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:32.052278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:32.052278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:32.052278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.052278 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:32.052862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:32.052862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:32.052862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.052862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:32.052862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:32.052862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:32.052862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:32.052862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:32.052862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:32.052862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:32.052862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:32.052862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:32.052862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:32.052862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:32.053203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:32.053203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:32.053203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.053203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:32.053203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.053203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:32.053203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:32.053203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:32.053203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:32.053203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:32.053203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.053203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:32.053203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:32.053203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:32.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:32.053733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:32.053733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:32.053733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:32.053733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:32.053733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:32.053733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:32.053733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:32.053733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:32.053733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:32.053733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.053733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:32.284502 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:32.284777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:32.284867 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:32.284997 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:32.285336 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:32.285570 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:32.285700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.286156 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:32.286156 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:32.286156 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:32.286156 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:32.286156 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:32.286156 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:32.286156 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.286156 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:32.286935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.286935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:32.286935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:32.286935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:32.286935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.286935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:32.286935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:32.286935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.286935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:32.286935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:32.286935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:32.286935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.287280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:32.287280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:32.287280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:32.287280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:32.287280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:32.287280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:32.287280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:32.287280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:32.287280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:32.287280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:32.287280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:32.287280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:32.287280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:32.287642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.287642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:32.288034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.288034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:32.288034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:32.288034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:32.288034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:32.288034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:32.288034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.288034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:32.288034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:32.288034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.288034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:32.288034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:32.288034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.288034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:32.288034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:32.288034 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:32.288843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:32.289646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:32.289646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.289646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:32.297183 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:32.297434 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:32.297587 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:32.298238 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:32.323945 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:32.324177 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.324632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:32.324632 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:32.324632 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:32.324632 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:32.324632 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:32.324632 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:32.324632 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.324632 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:32.325208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.325208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:32.325208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:32.325208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:32.325208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.325208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:32.325208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:32.325208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.325208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:32.325208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:32.325208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:32.325208 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.325447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:32.325447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:32.325447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:32.325447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:32.325447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:32.325447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:32.325447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:32.325447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:32.325447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:32.325447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:32.325447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:32.325447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:32.325447 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:32.325683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.325683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:32.325981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.325981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:32.325981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:32.325981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:32.325981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:32.325981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:32.325981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.325981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:32.325981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:32.325981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.325981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:32.325981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:32.325981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.325981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:32.325981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:32.325981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:32.326835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:32.327240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:32.327240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:32.327240 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:36.973220 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:36.973759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:36.973759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:36.974234 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:36.974391 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:36.974505 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:36.974633 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:36.975042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:36.975042 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:36.975042 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:36.975042 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:36.975042 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:36.975042 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:36.975042 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:36.975042 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:36.975692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:36.975692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:36.975692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:36.975692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:36.975692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:36.975692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:36.975692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:36.975692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:36.975692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:36.975692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:36.975692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:36.975692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:36.975952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:36.975952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:36.975952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:36.975952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:36.975952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:36.975952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:36.975952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:36.975952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:36.975952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:36.975952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:36.975952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:36.975952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:36.975952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:36.976197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:36.976197 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:36.976519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:36.976519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:36.976519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:36.976519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:36.976519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:36.976519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:36.976519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:36.976519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:36.976519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:36.976519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:36.976519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:36.976519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:36.976519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:36.976519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:36.976519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:36.976519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:36.977343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:36.977728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:36.977728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:36.977728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:37.029650 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:37.030135 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:37.030275 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:37.030586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:50:37.031020 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:37.031241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.031901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:37.031901 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:37.031901 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:37.031901 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:37.031901 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:37.031901 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:37.031901 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.031901 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:37.032552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.032552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:37.032552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:37.032552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:37.032552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.032552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:37.032552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:37.032552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:37.032552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.032552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:37.032552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:37.032552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:37.032552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:37.032552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:37.032552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:37.032883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:37.032883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:37.032883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:37.032883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:37.032883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:37.032883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:37.032883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:37.032883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.032883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:37.033316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.033316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:37.033316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:37.033316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:37.033316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.033316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:37.033316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:37.033316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:50:37.033316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:37.033316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:37.033316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.033316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:37.033952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:37.033952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:37.033952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.033952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:37.033952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:37.033952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:37.033952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:37.033952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:37.033952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:37.033952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:37.033952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:37.033952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:37.033952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:37.033952 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:37.034239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:37.034239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:37.034239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.034239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:37.034239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.034239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:37.034239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:37.034239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:37.034239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:37.034239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:37.034239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.034239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:37.034239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:37.034239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:37.034524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:37.034912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:37.034912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:37.034912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:37.034912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:37.034912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:37.034912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:37.034912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:37.034912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:37.034912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:37.034912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:37.034912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.034912 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:37.050268 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:37.050579 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:37.050703 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:37.050825 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:37.051370 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:37.051669 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.052184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:37.052184 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:37.052184 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:37.052184 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:37.052184 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:37.052184 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:37.052184 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.052184 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:37.052905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.052905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:37.052905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:37.052905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:37.052905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.052905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:37.052905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:37.052905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:37.052905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.052905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:37.052905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:37.052905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:37.052905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:37.052905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:37.052905 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:37.053205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:37.053205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:37.053205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:37.053205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:37.053205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:37.053205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:37.053205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:37.053205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.053205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:37.053654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.053654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:37.053654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:37.053654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:37.053654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.053654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:37.053654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:37.053654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:37.053654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:37.053654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.053654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:37.054282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:37.054282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:37.054282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.054282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:37.054282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:37.054282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:37.054282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:37.054282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:37.054282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:37.054282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:37.054282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:37.054282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:37.054282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:37.054282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:37.054910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:37.054910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:37.054910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.054910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:37.054910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.054910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:37.054910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:37.054910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:37.054910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:37.054910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:37.054910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.054910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:37.054910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:37.054910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:37.055184 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:37.055577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:37.055577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:37.055577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:37.055577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:37.055577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:37.055577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:37.055577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:37.055577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:37.055577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:37.055577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.055577 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:37.284943 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:37.285198 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:37.285305 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:37.285462 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:37.286012 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:37.286267 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:37.286429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.287042 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:37.287042 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:37.287042 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:37.287042 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:37.287042 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:37.287042 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:37.287042 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.287042 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:37.287889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.287889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:37.287889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:37.287889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:37.287889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.287889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:37.287889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:37.287889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.287889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:37.287889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:37.287889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:37.287889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.288113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:37.288113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:37.288113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:37.288113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:37.288113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:37.288113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:37.288113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:37.288113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:37.288113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:37.288113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:37.288113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:37.288113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:37.288113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:37.288296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.288296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:37.288781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.288781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:37.288781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:37.288781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:37.288781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:37.288781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:37.288781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.288781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:37.288781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:37.288781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.288781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:37.288781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:37.288781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.288781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:37.288781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:37.288781 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:37.289085 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:37.289370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:37.289370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.289370 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:37.297459 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:37.297717 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:37.297821 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:37.298467 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:37.325801 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:37.326032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.326569 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:37.326569 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:37.326569 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:37.326569 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:37.326569 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:37.326569 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:37.326569 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.326569 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:37.327283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.327283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:37.327283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:37.327283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:37.327283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.327283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:37.327283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:37.327283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.327283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:37.327283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:37.327283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:37.327283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.327526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:37.327526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:37.327526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:37.327526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:37.327526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:37.327526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:37.327526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:37.327526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:37.327526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:37.327526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:37.327526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:37.327526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:37.327526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:37.327755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.327755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:37.328046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.328046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:37.328046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:37.328046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:37.328046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:37.328046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:37.328046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.328046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:37.328046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:37.328046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.328046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:37.328046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:37.328046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.328046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:37.328046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:37.328046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:37.328313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:37.328728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:37.328728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:37.328728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:41.973730 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:41.974309 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:41.974309 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:41.974731 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:41.974941 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:41.975083 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:41.975191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:41.975678 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:41.975678 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:41.975678 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:41.975678 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:41.975678 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:41.975678 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:41.975678 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:41.975678 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:41.976383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:41.976383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:41.976383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:41.976383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:41.976383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:41.976383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:41.976383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:41.976383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:41.976383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:41.976383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:41.976383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:41.976383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:41.976868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:41.976868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:41.976868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:41.976868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:41.976868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:41.976868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:41.976868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:41.976868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:41.976868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:41.976868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:41.976868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:41.976868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:41.976868 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:41.977349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:41.977349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:41.977349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:41.977349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:41.977349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:41.977349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:41.977349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:41.977349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:41.977349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:41.977349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:41.977349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:41.977349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:41.977349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:41.977349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:41.977349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:41.977894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:41.978215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:41.978215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:41.978215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:41.978215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:41.978215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:41.978215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:42.030103 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:42.030319 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:42.030439 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:42.030586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:50:42.031060 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:42.031232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.031683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:42.031683 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:42.031683 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:42.031683 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:42.031683 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:42.031683 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:42.031683 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.031683 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:42.032256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.032256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:42.032256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:42.032256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:42.032256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.032256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:42.032256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:42.032256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:42.032256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.032256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:42.032256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:42.032256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:42.032256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:42.032256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:42.032256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:42.032618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:42.032618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:42.032618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:42.032618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:42.032618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:42.032618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:42.032618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:42.032618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.032618 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:42.032864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.032864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:42.032864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:42.032864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:42.032864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.032864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:42.032864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:42.032864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:50:42.032864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:42.032864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:42.032864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.032864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:42.033080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:42.033080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:42.033080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.033080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:42.033080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:42.033080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:42.033080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:42.033080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:42.033080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:42.033080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:42.033080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:42.033080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:42.033080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:42.033080 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:42.033303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:42.033303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:42.033303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.033303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:42.033522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.033522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:42.033522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:42.033522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:42.033522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:42.033522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:42.033522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.033522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:42.033522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:42.033522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.033522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:42.033522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:42.033522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.033522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:42.033522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:42.033522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:42.033806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:42.034088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:42.034088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:42.034088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:42.034088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:42.034088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:42.034088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.034088 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:42.050590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:42.050772 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:42.050859 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:42.050958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:42.051421 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:42.051667 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.052111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:42.052111 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:42.052111 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:42.052111 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:42.052111 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:42.052111 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:42.052111 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.052111 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:42.052687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.052687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:42.052687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:42.052687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:42.052687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.052687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:42.052687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:42.052687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:42.052687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.052687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:42.052687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:42.052687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:42.052687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:42.052687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:42.052687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:42.053056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:42.053056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:42.053056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:42.053056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:42.053056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:42.053056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:42.053056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:42.053056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.053056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:42.053300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.053300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:42.053300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:42.053300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:42.053300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.053300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:42.053300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:42.053300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:42.053300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:42.053300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.053300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:42.053518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:42.053518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:42.053518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.053518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:42.053518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:42.053518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:42.053518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:42.053518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:42.053518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:42.053518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:42.053518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:42.053518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:42.053518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:42.053518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:42.053765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:42.053765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:42.053765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.053765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.053998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:42.054287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:42.054611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:42.054611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:42.054611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.054611 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:42.285365 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:42.285627 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:42.285736 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:42.285874 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:42.286167 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:42.286352 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:42.286507 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.286900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:42.286900 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:42.286900 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:42.286900 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:42.286900 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:42.286900 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:42.286900 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.286900 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:42.287525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.287525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:42.287525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:42.287525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:42.287525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.287525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:42.287525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:42.287525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.287525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:42.287525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:42.287525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:42.287525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.287777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:42.287777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:42.287777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:42.287777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:42.287777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:42.287777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:42.287777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:42.287777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:42.287777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:42.287777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:42.287777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:42.287777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:42.287777 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:42.287993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.287993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:42.288241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.288241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:42.288241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:42.288241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:42.288241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:42.288241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:42.288241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.288241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:42.288241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:42.288241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.288241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:42.288241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:42.288241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.288241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:42.288241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:42.288241 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:42.288558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:42.288869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:42.288869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.288869 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:42.297764 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:42.298011 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:42.298084 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:42.298701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:42.323875 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:42.324025 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.324419 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:42.324419 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:42.324419 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:42.324419 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:42.324419 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:42.324419 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:42.324419 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.324419 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:42.325147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.325147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:42.325147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:42.325147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:42.325147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.325147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:42.325147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:42.325147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.325147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:42.325147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:42.325147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:42.325147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.325595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:42.325595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:42.325595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:42.325595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:42.325595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:42.325595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:42.325595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:42.325595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:42.325595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:42.325595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:42.325595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:42.325595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:42.325595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:42.325992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.325992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:42.326213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.326213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:42.326213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:42.326213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:42.326213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:42.326213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:42.326213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.326213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:42.326213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:42.326213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.326213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:42.326213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:42.326213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.326213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:42.326213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:42.326213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:42.326532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:42.326834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:42.326834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:42.326834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:46.974066 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:46.974695 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:46.974743 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:46.975527 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:46.975752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:46.975873 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:46.975983 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:46.976491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:46.976491 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:46.976491 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:46.976491 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:46.976491 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:46.976491 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:46.976491 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:46.976491 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:46.977166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:46.977166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:46.977166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:46.977166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:46.977166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:46.977166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:46.977166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:46.977166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:46.977166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:46.977166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:46.977166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:46.977166 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:46.977683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:46.977683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:46.977683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:46.977683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:46.977683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:46.977683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:46.977683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:46.977683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:46.977683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:46.977683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:46.977683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:46.977683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:46.977683 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:46.977973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:46.977973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:46.977973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:46.977973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:46.977973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:46.977973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:46.977973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:46.977973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:46.977973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:46.977973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:46.977973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:46.977973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:46.977973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:46.977973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:46.977973 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:46.978341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:46.978798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:46.978798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:46.978798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:46.978798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:46.978798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:46.978798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:47.030478 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:47.030874 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:47.030968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:47.031089 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:50:47.031395 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:47.031609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.032022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:47.032022 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:47.032022 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:47.032022 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:47.032022 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:47.032022 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:47.032022 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.032022 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:47.032627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.032627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:47.032627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:47.032627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:47.032627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.032627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:47.032627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:47.032627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:47.032627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.032627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:47.032627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:47.032627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:47.032627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:47.032627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:47.032627 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:47.032907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:47.032907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:47.032907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:47.032907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:47.032907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:47.032907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:47.032907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:47.032907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.032907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:47.033322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.033322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:47.033322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:47.033322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:47.033322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.033322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:47.033322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:47.033322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:50:47.033322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:47.033322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:47.033322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.033322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:47.033829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:47.033829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:47.033829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.033829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:47.033829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:47.033829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:47.033829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:47.033829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:47.033829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:47.033829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:47.033829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:47.033829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:47.033829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:47.033829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:47.034254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:47.034254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:47.034254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.034254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:47.034254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.034254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:47.034254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:47.034254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:47.034254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:47.034254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:47.034254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.034254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:47.034254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:47.034254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:47.034487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:47.034790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:47.034790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:47.034790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:47.034790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:47.034790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:47.034790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:47.034790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:47.034790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:47.034790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:47.034790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:47.034790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.034790 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:47.050931 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:47.051267 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:47.051382 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:47.051529 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:47.051925 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:47.052121 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.052561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:47.052561 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:47.052561 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:47.052561 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:47.052561 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:47.052561 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:47.052561 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.052561 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:47.053160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.053160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:47.053160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:47.053160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:47.053160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.053160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:47.053160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:47.053160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:47.053160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.053160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:47.053160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:47.053160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:47.053160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:47.053160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:47.053160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:47.053495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:47.053495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:47.053495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:47.053495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:47.053495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:47.053495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:47.053495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:47.053495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.053495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:47.053842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.053842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:47.053842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:47.053842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:47.053842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.053842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:47.053842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:47.053842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:47.053842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:47.053842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.053842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:47.054031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:47.054031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:47.054031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.054031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:47.054031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:47.054031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:47.054031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:47.054031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:47.054031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:47.054031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:47.054031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:47.054031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:47.054031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:47.054031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:47.054327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:47.054327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:47.054327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.054327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:47.054857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:47.055191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:47.055191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:47.055191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.055191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:47.285764 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:47.286105 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:47.286219 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:47.286359 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:47.286851 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:47.287158 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:47.287313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.287788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:47.287788 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:47.287788 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:47.287788 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:47.287788 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:47.287788 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:47.287788 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.287788 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:47.288458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.288458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:47.288458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:47.288458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:47.288458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.288458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:47.288458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:47.288458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.288458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:47.288458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:47.288458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:47.288458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.288733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:47.288733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:47.288733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:47.288733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:47.288733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:47.288733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:47.288733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:47.288733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:47.288733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:47.288733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:47.288733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:47.288733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:47.288733 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:47.289112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.289112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:47.289198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.289198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:47.289198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:47.289198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:47.289198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:47.289198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:47.289198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.289198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:47.289198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:47.289198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.289198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:47.289198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:47.289198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.289198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:47.289198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:47.289198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:47.289508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:47.289885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:47.289885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.289885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:47.298095 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:47.298305 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:47.298393 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:47.298995 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:47.325534 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:47.325860 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.326398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:47.326398 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:47.326398 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:47.326398 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:47.326398 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:47.326398 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:47.326398 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.326398 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:47.327141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.327141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:47.327141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:47.327141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:47.327141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.327141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:47.327141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:47.327141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.327141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:47.327141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:47.327141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:47.327141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.327371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:47.327371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:47.327371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:47.327371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:47.327371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:47.327371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:47.327371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:47.327371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:47.327371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:47.327371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:47.327371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:47.327371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:47.327371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:47.327626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.327626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:47.327908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.327908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:47.327908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:47.327908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:47.327908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:47.327908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:47.327908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.327908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:47.327908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:47.327908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.327908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:47.327908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:47.327908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.327908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:47.327908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:47.327908 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:47.328201 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:47.328500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:47.328500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:47.328500 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:51.974500 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:51.975101 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:51.975101 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:51.975366 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:51.975513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:51.975653 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:51.975756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:51.976154 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:51.976154 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:51.976154 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:51.976154 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:51.976154 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:51.976154 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:51.976154 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:51.976154 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:51.976789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:51.976789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:51.976789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:51.976789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:51.976789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:51.976789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:51.976789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:51.976789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:51.976789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:51.976789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:51.976789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:51.976789 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:51.977149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:51.977149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:51.977149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:51.977149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:51.977149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:51.977149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:51.977149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:51.977149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:51.977149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:51.977149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:51.977149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:51.977149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:51.977149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:51.977489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:51.977489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:51.977825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:51.977825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:51.977825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:51.977825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:51.977825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:51.977825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:51.977825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:51.977825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:51.977825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:51.977825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:51.977825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:51.977825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:51.977825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:51.977825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:51.977825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:51.977825 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:51.978182 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:51.978589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:51.978589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:51.978589 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:52.030958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:52.031213 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:52.031334 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:52.031489 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:50:52.031818 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:52.031990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.032417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:52.032417 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:52.032417 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:52.032417 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:52.032417 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:52.032417 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:52.032417 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.032417 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:52.033124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.033124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:52.033124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:52.033124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:52.033124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.033124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:52.033124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:52.033124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:52.033124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.033124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:52.033124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:52.033124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:52.033124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:52.033124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:52.033124 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:52.033647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:52.033647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:52.033647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:52.033647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:52.033647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:52.033647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:52.033647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:52.033647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.033647 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:52.034024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.034024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:52.034024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:52.034024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:52.034024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.034024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:52.034024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:52.034024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:50:52.034024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:52.034024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:52.034024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.034024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:52.034503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:52.034503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:52.034503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.034503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:52.034503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:52.034503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:52.034503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:52.034503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:52.034503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:52.034503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:52.034503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:52.034503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:52.034503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:52.034503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:52.034947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:52.034947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:52.034947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.034947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:52.035168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.035168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:52.035168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:52.035168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:52.035168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:52.035168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:52.035168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.035168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:52.035168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:52.035168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.035168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:52.035168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:52.035168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.035168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:52.035168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:52.035168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:52.035557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:52.035873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:52.035873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:52.035873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:52.035873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:52.035873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:52.035873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.035873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:52.051394 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:52.051821 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:52.051940 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:52.052082 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:52.052430 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:52.052676 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.053094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:52.053094 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:52.053094 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:52.053094 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:52.053094 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:52.053094 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:52.053094 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.053094 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:52.053784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.053784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:52.053784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:52.053784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:52.053784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.053784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:52.053784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:52.053784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:52.053784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.053784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:52.053784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:52.053784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:52.053784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:52.053784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:52.053784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:52.054213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:52.054213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:52.054213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:52.054213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:52.054213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:52.054213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:52.054213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:52.054213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.054213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:52.054696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.054696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:52.054696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:52.054696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:52.054696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.054696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:52.054696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:52.054696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:52.054696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:52.054696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.054696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:52.055047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:52.055047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:52.055047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.055047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:52.055047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:52.055047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:52.055047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:52.055047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:52.055047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:52.055047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:52.055047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:52.055047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:52.055047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:52.055047 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:52.055437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:52.055437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:52.055437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.055437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.056055 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:52.056383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:52.056735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:52.056735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:52.056735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.056735 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:52.286147 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:52.286526 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:52.286635 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:52.286784 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:52.287136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:52.287374 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:52.287604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.288100 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:52.288100 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:52.288100 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:52.288100 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:52.288100 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:52.288100 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:52.288100 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.288100 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:52.288775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.288775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:52.288775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:52.288775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:52.288775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.288775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:52.288775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:52.288775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.288775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:52.288775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:52.288775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:52.288775 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.289064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:52.289064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:52.289064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:52.289064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:52.289064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:52.289064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:52.289064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:52.289064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:52.289064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:52.289064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:52.289064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:52.289064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:52.289064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:52.289284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.289284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:52.289413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.289413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:52.289413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:52.289413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:52.289413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:52.289413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:52.289413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.289413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:52.289413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:52.289413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.289413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:52.289413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:52.289413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.289413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:52.289413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:52.289413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:52.289742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:52.290039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:52.290039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.290039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:52.298495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:52.298745 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:52.298886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:52.299398 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:52.326164 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:52.326341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.326903 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:52.326903 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:52.326903 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:52.326903 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:52.326903 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:52.326903 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:52.326903 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.326903 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:52.327629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.327629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:52.327629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:52.327629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:52.327629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.327629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:52.327629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:52.327629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.327629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:52.327629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:52.327629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:52.327629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.327851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:52.327851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:52.327851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:52.327851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:52.327851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:52.327851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:52.327851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:52.327851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:52.327851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:52.327851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:52.327851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:52.327851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:52.327851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:52.328163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.328163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:52.328437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.328437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:52.328437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:52.328437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:52.328437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:52.328437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:52.328437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.328437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:52.328437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:52.328437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.328437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:52.328437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:52.328437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.328437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:52.328437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:52.328437 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:52.328778 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:52.329173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:52.329173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:52.329173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:56.974969 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:56.975397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:56.975397 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:56.975812 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:56.975974 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:56.976104 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:56.976226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:56.976701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:56.976701 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:56.976701 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:56.976701 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:56.976701 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:56.976701 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:56.976701 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:56.976701 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:56.977660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:56.977660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:56.977660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:56.977660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:56.977660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:56.977660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:56.977660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:56.977660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:56.977660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:56.977660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:56.977660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:56.977660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:56.977865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:56.977865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:56.977865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:56.977865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:56.977865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:56.977865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:56.977865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:56.977865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:56.977865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:56.977865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:56.977865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:56.977865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:56.977865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:56.978060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:56.978060 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:56.978751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:56.978751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:56.978751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:56.978751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:56.978751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:56.978751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:56.978751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:56.978751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:56.978751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:56.978751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:56.978751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:56.978751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:56.978751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:56.978751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:56.978751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:56.978751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:56.979022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:56.979287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:56.979287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:56.979287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:57.031424 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:57.031786 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:57.031926 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:57.032063 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:50:57.032402 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:57.032616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.033062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:57.033062 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:57.033062 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:57.033062 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:57.033062 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:57.033062 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:57.033062 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.033062 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:57.033883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.033883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:57.033883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:57.033883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:57.033883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.033883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:57.033883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:57.033883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:57.033883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.033883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:57.033883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:57.033883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:57.033883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:57.033883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:57.033883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:57.034130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:57.034130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:57.034130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:57.034130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:57.034130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:57.034130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:57.034130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:57.034130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.034130 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:57.034811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.034811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:57.034811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:57.034811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:57.034811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.034811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:57.034811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:57.034811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:50:57.034811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:57.034811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:57.034811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.034811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:57.035114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:57.035114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:57.035114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.035114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:57.035114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:57.035114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:57.035114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:57.035114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:57.035114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:57.035114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:57.035114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:57.035114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:57.035114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:57.035114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:57.035458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:57.035458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:57.035458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.035458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:57.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:57.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:57.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:57.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:57.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:57.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:57.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:57.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:57.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:57.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:57.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:57.035933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:57.036181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:57.036456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:57.036456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:57.036456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:57.036456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:57.036456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:57.036456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.036456 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:57.051948 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:57.052262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:57.052421 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:57.052608 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:57.052954 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:57.053185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.053728 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:57.053728 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:57.053728 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:57.053728 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:57.053728 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:57.053728 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:57.053728 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.053728 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:57.054480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.054480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:50:57.054480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:57.054480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:57.054480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.054480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:57.054480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:57.054480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:50:57.054480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.054480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:50:57.054480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:57.054480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:50:57.054480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:57.054480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:50:57.054480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:50:57.054774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:57.054774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:50:57.054774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:57.054774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:50:57.054774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:50:57.054774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:57.054774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:50:57.054774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.054774 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:50:57.055423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.055423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:57.055423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:57.055423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:57.055423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.055423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:57.055423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:57.055423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:57.055423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:57.055423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.055423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:57.055675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:57.055675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:57.055675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.055675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:57.055675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:57.055675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:57.055675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:57.055675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:57.055675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:57.055675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:57.055675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:57.055675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:57.055675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:57.055675 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:57.055928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:57.055928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:57.055928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.055928 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.056716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:57.056991 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:57.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:57.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:57.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:57.286608 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:57.286943 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:57.287079 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:57.287274 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:57.287866 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:57.288191 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:57.288377 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.288988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:57.288988 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:57.288988 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:57.288988 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:57.288988 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:57.288988 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:57.288988 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.288988 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:57.289848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.289848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:57.289848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:57.289848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:57.289848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.289848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:57.289848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:57.289848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.289848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:57.289848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:57.289848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:57.289848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.290193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:57.290193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:57.290193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:57.290193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:57.290193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:57.290193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:57.290193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:57.290193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:57.290193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:57.290193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:57.290193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:57.290193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:57.290193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:57.290517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.290517 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:57.291011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.291011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:57.291011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:57.291011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:57.291011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:57.291011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:57.291011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.291011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:57.291011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:57.291011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.291011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:57.291011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:57.291011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.291011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:57.291011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:57.291011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:57.291350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:57.291662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:57.291662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.291662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:50:57.298952 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:50:57.299171 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:50:57.299325 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:50:57.300061 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:50:57.328446 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:50:57.328754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.329443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:50:57.329443 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:57.329443 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:50:57.329443 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:50:57.329443 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:57.329443 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:50:57.329443 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.329443 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:50:57.330305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.030 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.330305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:50:57.330305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:57.330305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:50:57.330305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.330305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:57.330305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:57.330305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.330305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:57.330305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:57.330305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:50:57.330305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.330605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:50:57.330605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:57.330605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:50:57.330605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:57.330605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:50:57.330605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:50:57.330605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:57.330605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:50:57.330605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:57.330605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:50:57.330605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:50:57.330605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:57.330605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:50:57.330808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.330808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:50:57.331165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.331165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:50:57.331165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:50:57.331165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:50:57.331165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:50:57.331165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:50:57.331165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.331165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:50:57.331165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:50:57.331165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.331165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:50:57.331165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:50:57.331165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.331165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:50:57.331165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:50:57.331165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:50:57.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:50:57.332322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:50:57.332322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:50:57.332322 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:01.975376 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:01.975945 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:01.975945 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:01.976486 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:01.976651 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:01.976775 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:01.976885 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:01.977320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:01.977320 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:01.977320 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:01.977320 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:01.977320 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:01.977320 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:01.977320 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:01.977320 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:01.978282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:01.978282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:01.978282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:01.978282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:01.978282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:01.978282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:01.978282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:01.978282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:01.978282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:01.978282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:01.978282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:01.978282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:01.978813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:01.978813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:01.978813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:01.978813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:01.978813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:01.978813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:01.978813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:01.978813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:01.978813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:01.978813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:01.978813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:01.978813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:01.978813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:01.979125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:01.979125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:01.979239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:01.979239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:01.979239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:01.979239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:01.979239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:01.979239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:01.979239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:01.979239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:01.979239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:01.979239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:01.979239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:01.979239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:01.979239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:01.979239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:01.979239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:01.979239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:01.979714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:01.980024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:01.980024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:01.980024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:02.031856 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:02.032033 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:02.032138 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:02.032204 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:51:02.032590 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:02.032744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.033131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:02.033131 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:02.033131 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:02.033131 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:02.033131 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:02.033131 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:02.033131 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.033131 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:02.033698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.033698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:02.033698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:02.033698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:02.033698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.033698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:02.033698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:02.033698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:02.033698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.033698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:02.033698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:02.033698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:02.033698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:02.033698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:02.033698 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:02.033954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:02.033954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:02.033954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:02.033954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:02.033954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:02.033954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:02.033954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:02.033954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.033954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:02.034325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.034325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:02.034325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:02.034325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:02.034325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.034325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:02.034325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:02.034325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:51:02.034325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:02.034325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:02.034325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.034325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:02.034998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:02.034998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:02.034998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.034998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:02.034998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:02.034998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:02.034998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:02.034998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:02.034998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:02.034998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:02.034998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:02.034998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:02.034998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:02.034998 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:02.035360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:02.035360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:02.035360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.035360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:02.035360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.035360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:02.035360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:02.035360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:02.035360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:02.035360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:02.035360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.035360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:02.035360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:02.035360 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:02.035620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:02.035901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:02.035901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:02.035901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:02.035901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:02.035901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:02.035901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:02.035901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:02.035901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:02.035901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:02.035901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:02.035901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.035901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:02.052348 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:02.052579 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:02.052661 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:02.052776 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:02.053143 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:02.053420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.053999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:02.053999 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:02.053999 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:02.053999 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:02.053999 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:02.053999 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:02.053999 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.053999 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:02.054581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.054581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:02.054581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:02.054581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:02.054581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.054581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:02.054581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:02.054581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:02.054581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.054581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:02.054581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:02.054581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:02.054581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:02.054581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:02.054581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:02.054827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:02.054827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:02.054827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:02.054827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:02.054827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:02.054827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:02.054827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:02.054827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.054827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:02.055147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.055147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:02.055147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:02.055147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:02.055147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.055147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:02.055147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:02.055147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:02.055147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:02.055147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.055147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:02.055591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:02.055591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:02.055591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.055591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:02.055591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:02.055591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:02.055591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:02.055591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:02.055591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:02.055591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:02.055591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:02.055591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:02.055591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:02.055591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:02.056103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:02.056103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:02.056103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.056103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:02.056103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.056103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:02.056103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:02.056103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:02.056103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:02.056103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:02.056103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.056103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:02.056103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:02.056103 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:02.056320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:02.056630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:02.056630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:02.056630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:02.056630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:02.056630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:02.056630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:02.056630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:02.056630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:02.056630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:02.056630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.056630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:02.287118 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:02.287419 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:02.287591 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:02.287783 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:02.288080 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:02.288284 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:02.288411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.288875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:02.288875 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:02.288875 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:02.288875 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:02.288875 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:02.288875 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:02.288875 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.288875 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:02.289859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.289859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:02.289859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:02.289859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:02.289859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.289859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:02.289859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:02.289859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.289859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:02.289859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:02.289859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:02.289859 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.290082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:02.290082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:02.290082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:02.290082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:02.290082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:02.290082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:02.290082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:02.290082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:02.290082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:02.290082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:02.290082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:02.290082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:02.290082 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:02.290287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.290287 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:02.290966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.290966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:02.290966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:02.290966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:02.290966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:02.290966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:02.290966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.290966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:02.290966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:02.290966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.290966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:02.290966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:02.290966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.290966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:02.290966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:02.290966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:02.291237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:02.291572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:02.291572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.291572 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:02.299334 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:02.299674 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:02.299805 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:02.300684 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:02.333492 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:02.333670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.334078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:02.334078 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:02.334078 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:02.334078 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:02.334078 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:02.334078 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:02.334078 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.334078 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:02.334788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.334788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:02.334788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:02.334788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:02.334788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.334788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:02.334788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:02.334788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.334788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:02.334788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:02.334788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:02.334788 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.335418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:02.335418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:02.335418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:02.335418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:02.335418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:02.335418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:02.335418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:02.335418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:02.335418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:02.335418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:02.335418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:02.335418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:02.335418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:02.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:02.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:02.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:02.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:02.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:02.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:02.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:02.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:02.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:02.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:02.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:02.336112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:02.336397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:02.336397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:02.336397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:02.336397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:02.336397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:02.336397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:06.975778 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:06.976136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:06.976136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:06.976768 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:06.976944 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:06.977058 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:06.977183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:06.977664 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:06.977664 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:06.977664 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:06.977664 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:06.977664 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:06.977664 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:06.977664 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:06.977664 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:06.978380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:06.978380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:06.978380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:06.978380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:06.978380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:06.978380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:06.978380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:06.978380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:06.978380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:06.978380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:06.978380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:06.978380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:06.978657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:06.978657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:06.978657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:06.978657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:06.978657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:06.978657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:06.978657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:06.978657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:06.978657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:06.978657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:06.978657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:06.978657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:06.978657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:06.978884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:06.978884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:06.979187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:06.979187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:06.979187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:06.979187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:06.979187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:06.979187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:06.979187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:06.979187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:06.979187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:06.979187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:06.979187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:06.979187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:06.979187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:06.979187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:06.979187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:06.979187 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:06.979494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:06.979822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:06.979822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:06.979822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:07.032194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:07.032423 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:07.032530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:07.032655 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:51:07.033088 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:07.033259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.033731 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:07.033731 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:07.033731 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:07.033731 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:07.033731 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:07.033731 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:07.033731 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.033731 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:07.034341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.034341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:07.034341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:07.034341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:07.034341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.034341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:07.034341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:07.034341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:07.034341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.034341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:07.034341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:07.034341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:07.034341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:07.034341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:07.034341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:07.034671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:07.034671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:07.034671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:07.034671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:07.034671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:07.034671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:07.034671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:07.034671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.034671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:07.034992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.034992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:07.034992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:07.034992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:07.034992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.034992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:07.034992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:07.034992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:51:07.034992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:07.034992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:07.034992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.034992 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:07.035237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:07.035237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:07.035237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.035237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:07.035237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:07.035237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:07.035237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:07.035237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:07.035237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:07.035237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:07.035237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:07.035237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:07.035237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:07.035237 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:07.035476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:07.035476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:07.035476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.035476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:07.035665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.035665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:07.035665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:07.035665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:07.035665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:07.035665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:07.035665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.035665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:07.035665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:07.035665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.035665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:07.035665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:07.035665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.035665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:07.035665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:07.035665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:07.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:07.036218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:07.036218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:07.036218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:07.036218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:07.036218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:07.036218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.036218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:07.052741 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:07.052967 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:07.053088 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:07.053195 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:07.053556 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:07.053755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.054175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:07.054175 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:07.054175 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:07.054175 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:07.054175 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:07.054175 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:07.054175 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.054175 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:07.054782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.054782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:07.054782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:07.054782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:07.054782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.054782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:07.054782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:07.054782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:07.054782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.054782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:07.054782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:07.054782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:07.054782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:07.054782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:07.054782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:07.055128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:07.055128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:07.055128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:07.055128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:07.055128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:07.055128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:07.055128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:07.055128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.055128 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:07.055350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.055350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:07.055350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:07.055350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:07.055350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.055350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:07.055350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:07.055350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:07.055350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:07.055350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.055350 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:07.055596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:07.055596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:07.055596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.055596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:07.055596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:07.055596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:07.055596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:07.055596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:07.055596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:07.055596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:07.055596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:07.055596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:07.055596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:07.055596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:07.055831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:07.055831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:07.055831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.055831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.056054 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:07.056352 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:07.056679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:07.056679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:07.056679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.056679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:07.287622 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:07.287819 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:07.287928 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:07.288081 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:07.288369 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:07.288620 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:07.288738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.289155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:07.289155 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:07.289155 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:07.289155 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:07.289155 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:07.289155 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:07.289155 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.289155 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:07.289813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.289813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:07.289813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:07.289813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:07.289813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.289813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:07.289813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:07.289813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.289813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:07.289813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:07.289813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:07.289813 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.290226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:07.290226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:07.290226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:07.290226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:07.290226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:07.290226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:07.290226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:07.290226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:07.290226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:07.290226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:07.290226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:07.290226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:07.290226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:07.290634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.290634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:07.290634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.290634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:07.290634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:07.290634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:07.290634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:07.290634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:07.290634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.290634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:07.290634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:07.290634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.290634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:07.290634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:07.290634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:07.291073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:07.291670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:07.291670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:07.291670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:07.291670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:07.291670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.291670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:07.299812 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:07.300031 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:07.300112 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:07.300842 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:07.330234 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:07.330475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.331629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:07.331629 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:07.331629 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:07.331629 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:07.331629 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:07.331629 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:07.331629 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.331629 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:07.331925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.331925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:07.331925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:07.331925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:07.331925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.331925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:07.331925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:07.331925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.331925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:07.331925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:07.331925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:07.331925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.332395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:07.332395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:07.332395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:07.332395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:07.332395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:07.332395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:07.332395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:07.332395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:07.332395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:07.332395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:07.332395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:07.332395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:07.332395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:07.332831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.332831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:07.332831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.332831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:07.332831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:07.332831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:07.332831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:07.332831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:07.332831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.332831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:07.332831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:07.332831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.332831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:07.332831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:07.332831 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:07.333351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:07.333975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:07.333975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:07.333975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:07.333975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:07.333975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:07.333975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:11.976304 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:11.976800 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:11.976800 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:11.977387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:11.977642 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:11.977774 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:11.977923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:11.978433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:11.978433 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:11.978433 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:11.978433 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:11.978433 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:11.978433 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:11.978433 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:11.978433 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:11.979293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:11.979293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:11.979293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:11.979293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:11.979293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:11.979293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:11.979293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:11.979293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:11.979293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:11.979293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:11.979293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:11.979293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:11.980005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:11.980005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:11.980005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:11.980005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:11.980005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:11.980005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:11.980005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:11.980005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:11.980005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:11.980005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:11.980005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:11.980005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:11.980005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:11.980359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:11.980359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:11.980359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:11.980359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:11.980359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:11.980359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:11.980359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:11.980359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:11.980359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:11.980359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:11.980359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:11.980359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:11.980359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:11.980359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:11.980359 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:11.980865 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:11.981378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:11.981378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:11.981378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:11.981378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:11.981378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:11.981378 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:12.032654 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:12.032886 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:12.033000 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:12.033113 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:51:12.033605 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:12.033770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.034203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:12.034203 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:12.034203 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:12.034203 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:12.034203 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:12.034203 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:12.034203 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.034203 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:12.034848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.034848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:12.034848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:12.034848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:12.034848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.034848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:12.034848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:12.034848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:12.034848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.034848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:12.034848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:12.034848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:12.034848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:12.034848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:12.034848 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:12.035397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:12.035397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:12.035397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:12.035397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:12.035397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:12.035397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:12.035397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:12.035397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.035397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:12.035773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.035773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:12.035773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:12.035773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:12.035773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.035773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:12.035773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:12.035773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:51:12.035773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:12.035773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:12.035773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.035773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:12.036211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:12.036211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:12.036211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.036211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:12.036211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:12.036211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:12.036211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:12.036211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:12.036211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:12.036211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:12.036211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:12.036211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:12.036211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:12.036211 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:12.036695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:12.036695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:12.036695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.036695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:12.036695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.036695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:12.036695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:12.036695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:12.036695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:12.036695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:12.036695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.036695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:12.036695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:12.036695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:12.037181 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:12.038196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:12.038196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:12.038196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:12.038196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:12.038196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:12.038196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:12.038196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:12.038196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:12.038196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:12.038196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:12.038196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.038196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:12.053141 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:12.053387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:12.053591 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:12.053655 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:12.053975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:12.054207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.054658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:12.054658 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:12.054658 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:12.054658 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:12.054658 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:12.054658 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:12.054658 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.054658 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:12.055188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.055188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:12.055188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:12.055188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:12.055188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.055188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:12.055188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:12.055188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:12.055188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.055188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:12.055188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:12.055188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:12.055188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:12.055188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:12.055188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:12.055705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:12.055705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:12.055705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:12.055705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:12.055705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:12.055705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:12.055705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:12.055705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.055705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:12.056011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.056011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:12.056011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:12.056011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:12.056011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.056011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:12.056011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:12.056011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:12.056011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:12.056011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.056011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:12.056423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:12.056423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:12.056423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.056423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:12.056423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:12.056423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:12.056423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:12.056423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:12.056423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:12.056423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:12.056423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:12.056423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:12.056423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:12.056423 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:12.056921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:12.056921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:12.056921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.056921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:12.056921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.056921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:12.056921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:12.056921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:12.056921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:12.056921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:12.056921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.056921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:12.056921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:12.056921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:12.057394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:12.058029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:12.058029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:12.058029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:12.058029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:12.058029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:12.058029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:12.058029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:12.058029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:12.058029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:12.058029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.058029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:12.288013 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:12.288225 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:12.288444 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:12.288658 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:12.288953 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:12.289141 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:12.289271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.289803 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:12.289803 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:12.289803 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:12.289803 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:12.289803 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:12.289803 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:12.289803 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.289803 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:12.290552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.290552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:12.290552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:12.290552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:12.290552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.290552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:12.290552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:12.290552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.290552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:12.290552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:12.290552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:12.290552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.290769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:12.290769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:12.290769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:12.290769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:12.290769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:12.290769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:12.290769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:12.290769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:12.290769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:12.290769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:12.290769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:12.290769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:12.290769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:12.290975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.290975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:12.291484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.291484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:12.291484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:12.291484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:12.291484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:12.291484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:12.291484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.291484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:12.291484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:12.291484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.291484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:12.291484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:12.291484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.291484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:12.291484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:12.291484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:12.291799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:12.292209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:12.292209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.292209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:12.300130 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:12.300345 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:12.300463 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:12.301077 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:12.330189 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:12.330380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.330970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:12.330970 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:12.330970 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:12.330970 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:12.330970 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:12.330970 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:12.330970 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.330970 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:12.331842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.331842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:12.331842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:12.331842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:12.331842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.331842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:12.331842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:12.331842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.331842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:12.331842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:12.331842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:12.331842 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.332099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:12.332099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:12.332099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:12.332099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:12.332099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:12.332099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:12.332099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:12.332099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:12.332099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:12.332099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:12.332099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:12.332099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:12.332099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:12.332311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.332311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:12.332578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.332578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:12.332578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:12.332578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:12.332578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:12.332578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:12.332578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.332578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:12.332578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:12.332578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.332578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:12.332578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:12.332578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.332578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:12.332578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:12.332578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:12.332875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:12.333247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:12.333247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:12.333247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:16.976845 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:16.977244 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:16.977244 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:16.977832 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:16.978009 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:16.978130 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:16.978236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:16.978689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:16.978689 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:16.978689 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:16.978689 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:16.978689 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:16.978689 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:16.978689 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:16.978689 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:16.979297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:16.979297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:16.979297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:16.979297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:16.979297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:16.979297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:16.979297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:16.979297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:16.979297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:16.979297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:16.979297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:16.979297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:16.979692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:16.979692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:16.979692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:16.979692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:16.979692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:16.979692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:16.979692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:16.979692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:16.979692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:16.979692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:16.979692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:16.979692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:16.979692 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:16.980012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:16.980012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:16.980101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:16.980101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:16.980101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:16.980101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:16.980101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:16.980101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:16.980101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:16.980101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:16.980101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:16.980101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:16.980101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:16.980101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:16.980101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:16.980101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:16.980101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:16.980101 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:16.980404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:16.980754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:16.980754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:16.980754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:17.033038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:17.033386 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:17.033497 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:17.033613 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:51:17.033920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:17.034089 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.034495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:17.034495 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:17.034495 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:17.034495 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:17.034495 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:17.034495 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:17.034495 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.034495 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:17.035029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.035029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:17.035029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:17.035029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:17.035029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.035029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:17.035029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:17.035029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:17.035029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.035029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:17.035029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:17.035029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:17.035029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:17.035029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:17.035029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:17.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:17.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:17.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:17.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:17.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:17.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:17.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:17.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.035297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:17.035802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.035802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:17.035802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:17.035802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:17.035802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.035802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:17.035802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:17.035802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:51:17.035802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:17.035802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:17.035802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.035802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:17.036020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:17.036020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:17.036020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.036020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:17.036020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:17.036020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:17.036020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:17.036020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:17.036020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:17.036020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:17.036020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:17.036020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:17.036020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:17.036020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:17.036228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:17.036228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:17.036228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.036228 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:17.036687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.036687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:17.036687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:17.036687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:17.036687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:17.036687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:17.036687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.036687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:17.036687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:17.036687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.036687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:17.036687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:17.036687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.036687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:17.036687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:17.036687 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:17.036947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:17.037216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:17.037216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:17.037216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:17.037216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:17.037216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:17.037216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.037216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:17.053468 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:17.053751 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:17.053883 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:17.054012 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:17.054296 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:17.054511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.054940 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:17.054940 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:17.054940 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:17.054940 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:17.054940 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:17.054940 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:17.054940 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.054940 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:17.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:17.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:17.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:17.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:17.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:17.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:17.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:17.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:17.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:17.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:17.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:17.055429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:17.055717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:17.055717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:17.055717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:17.055717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:17.055717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:17.055717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:17.055717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:17.055717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.055717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:17.056023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.056023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:17.056023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:17.056023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:17.056023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.056023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:17.056023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:17.056023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:17.056023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:17.056023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.056023 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:17.056312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:17.056312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:17.056312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.056312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:17.056312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:17.056312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:17.056312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:17.056312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:17.056312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:17.056312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:17.056312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:17.056312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:17.056312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:17.056312 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:17.056586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:17.056586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:17.056586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.056586 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.056807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:17.057107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:17.057435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:17.057435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:17.057435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.057435 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:17.288376 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:17.288805 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:17.288912 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:17.289095 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:17.289560 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:17.289860 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:17.290009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.290587 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:17.290587 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:17.290587 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:17.290587 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:17.290587 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:17.290587 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:17.290587 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.290587 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:17.291243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.291243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:17.291243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:17.291243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:17.291243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.291243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:17.291243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:17.291243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.291243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:17.291243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:17.291243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:17.291243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.291742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:17.291742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:17.291742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:17.291742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:17.291742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:17.291742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:17.291742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:17.291742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:17.291742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:17.291742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:17.291742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:17.291742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:17.291742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:17.292139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.292139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:17.292139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.292139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:17.292139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:17.292139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:17.292139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:17.292139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:17.292139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.292139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:17.292139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:17.292139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.292139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:17.292139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:17.292139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:17.292701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:17.293233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:17.293233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:17.293233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:17.293233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:17.293233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.293233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:17.300495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:17.300740 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:17.300824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:17.301434 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:17.327924 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:17.328119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.328645 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:17.328645 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:17.328645 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:17.328645 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:17.328645 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:17.328645 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:17.328645 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.328645 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:17.329428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.329428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:17.329428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:17.329428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:17.329428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.329428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:17.329428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:17.329428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.329428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:17.329428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:17.329428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:17.329428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.329744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:17.329744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:17.329744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:17.329744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:17.329744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:17.329744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:17.329744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:17.329744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:17.329744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:17.329744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:17.329744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:17.329744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:17.329744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:17.329970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.329970 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:17.330204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.330204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:17.330204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:17.330204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:17.330204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:17.330204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:17.330204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.330204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:17.330204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:17.330204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.330204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:17.330204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:17.330204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.330204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:17.330204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:17.330204 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:17.330526 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:17.330843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:17.330843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:17.330843 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:21.977262 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:21.977759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:21.977759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:21.978479 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:21.978730 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:21.978865 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:21.978990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:21.979536 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:21.979536 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:21.979536 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:21.979536 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:21.979536 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:21.979536 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:21.979536 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:21.979536 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:21.980323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:21.980323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:21.980323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:21.980323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:21.980323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:21.980323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:21.980323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:21.980323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:21.980323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:21.980323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:21.980323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:21.980323 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:21.980619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:21.980619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:21.980619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:21.980619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:21.980619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:21.980619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:21.980619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:21.980619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:21.980619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:21.980619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:21.980619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:21.980619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:21.980619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:21.980835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:21.980835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:21.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:21.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:21.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:21.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:21.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:21.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:21.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:21.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:21.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:21.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:21.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:21.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:21.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:21.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:21.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:21.981242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:21.981699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:21.982131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:21.982131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:21.982131 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:22.033526 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:22.033930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:22.034071 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:22.034200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:51:22.034681 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:22.034883 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.035335 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:22.035335 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:22.035335 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:22.035335 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:22.035335 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:22.035335 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:22.035335 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.035335 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:22.036155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.036155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:22.036155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:22.036155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:22.036155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.036155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:22.036155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:22.036155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:22.036155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.036155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:22.036155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:22.036155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:22.036155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:22.036155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:22.036155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:22.036410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:22.036410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:22.036410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:22.036410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:22.036410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:22.036410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:22.036410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:22.036410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.036410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:22.036949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.036949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:22.036949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:22.036949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:22.036949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.036949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:22.036949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:22.036949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:51:22.036949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:22.036949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:22.036949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.036949 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:22.037193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:22.037193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:22.037193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.037193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:22.037193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:22.037193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:22.037193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:22.037193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:22.037193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:22.037193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:22.037193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:22.037193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:22.037193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:22.037193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:22.037417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:22.037417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:22.037417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.037417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:22.037872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.037872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:22.037872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:22.037872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:22.037872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:22.037872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:22.037872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.037872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:22.037872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:22.037872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.037872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:22.037872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:22.037872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.037872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:22.037872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:22.037872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:22.038155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:22.038481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:22.038481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:22.038481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:22.038481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:22.038481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:22.038481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.038481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:22.053911 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:22.054190 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:22.054298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:22.054420 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:22.054835 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:22.055043 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.055516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:22.055516 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:22.055516 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:22.055516 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:22.055516 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:22.055516 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:22.055516 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.055516 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:22.056282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.056282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:22.056282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:22.056282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:22.056282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.056282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:22.056282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:22.056282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:22.056282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.056282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:22.056282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:22.056282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:22.056282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:22.056282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:22.056282 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:22.056565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:22.056565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:22.056565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:22.056565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:22.056565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:22.056565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:22.056565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:22.056565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.056565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:22.057071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.057071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:22.057071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:22.057071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:22.057071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.057071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:22.057071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:22.057071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:22.057071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:22.057071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.057071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:22.057293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:22.057293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:22.057293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.057293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:22.057293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:22.057293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:22.057293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:22.057293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:22.057293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:22.057293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:22.057293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:22.057293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:22.057293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:22.057293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:22.057532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:22.057532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:22.057532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.057532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.057996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:22.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:22.058708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:22.058708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:22.058708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.058708 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:22.288830 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:22.289082 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:22.289150 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:22.289300 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:22.289832 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:22.290092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:22.290274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.290832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:22.290832 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:22.290832 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:22.290832 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:22.290832 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:22.290832 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:22.290832 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.290832 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:22.291501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.291501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:22.291501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:22.291501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:22.291501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.291501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:22.291501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:22.291501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.291501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:22.291501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:22.291501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:22.291501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.291911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:22.291911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:22.291911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:22.291911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:22.291911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:22.291911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:22.291911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:22.291911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:22.291911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:22.291911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:22.291911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:22.291911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:22.291911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:22.292309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.292309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:22.292309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.292309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:22.292309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:22.292309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:22.292309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:22.292309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:22.292309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.292309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:22.292309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:22.292309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.292309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:22.292309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:22.292309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:22.292772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:22.293152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:22.293152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:22.293152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:22.293152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:22.293152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.293152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:22.300786 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:22.300962 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:22.301045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:22.301758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:22.306446 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=239 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:51:22.332073 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:22.332319 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.332929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:22.332929 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:22.332929 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:22.332929 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:22.332929 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:22.332929 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:22.332929 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.332929 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:22.333553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.032 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.333553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:22.333553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:22.333553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:22.333553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.333553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:22.333553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:22.333553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.333553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:22.333553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:22.333553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:22.333553 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.333799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:22.333799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:22.333799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:22.333799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:22.333799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:22.333799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:22.333799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:22.333799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:22.333799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:22.333799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:22.333799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:22.333799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:22.333799 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:22.334012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.334012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:22.334238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.334238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:22.334238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:22.334238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:22.334238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:22.334238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:22.334238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.334238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:22.334238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:22.334238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.334238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:22.334238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:22.334238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.334238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:22.334238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:22.334238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:22.334532 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:22.334897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:22.334897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:22.334897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:26.977709 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:26.978619 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:26.978619 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:26.979049 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:26.979258 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:26.979387 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:26.979502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:26.979987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:26.979987 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:26.979987 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:26.979987 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:26.979987 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:26.979987 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:26.979987 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:26.979987 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:26.980679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:26.980679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:26.980679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:26.980679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:26.980679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:26.980679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:26.980679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:26.980679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:26.980679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:26.980679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:26.980679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:26.980679 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:26.980919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:26.980919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:26.980919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:26.980919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:26.980919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:26.980919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:26.980919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:26.980919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:26.980919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:26.980919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:26.980919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:26.980919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:26.980919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:26.981143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:26.981143 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:26.981355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:26.981355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:26.981355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:26.981355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:26.981355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:26.981355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:26.981355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:26.981355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:26.981355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:26.981355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:26.981355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:26.981355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:26.981355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:26.981355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:26.981355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:26.981355 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:26.981663 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:26.981980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:26.981980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:26.981980 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:27.033938 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:27.034158 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:27.034247 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:27.034342 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:51:27.034826 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:27.034997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.035413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:27.035413 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:27.035413 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:27.035413 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:27.035413 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:27.035413 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:27.035413 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.035413 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:27.036051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.036051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:27.036051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:27.036051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:27.036051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.036051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:27.036051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:27.036051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:27.036051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.036051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:27.036051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:27.036051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:27.036051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:27.036051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:27.036051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:27.036301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:27.036301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:27.036301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:27.036301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:27.036301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:27.036301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:27.036301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:27.036301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.036301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:27.036662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.036662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:27.036662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:27.036662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:27.036662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.036662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:27.036662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:27.036662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:51:27.036662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:27.036662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:27.036662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.036662 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:27.036889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:27.036889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:27.036889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.036889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:27.036889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:27.036889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:27.036889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:27.036889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:27.036889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:27.036889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:27.036889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:27.036889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:27.036889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:27.036889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:27.037099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:27.037099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:27.037099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.037099 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:27.037316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.037316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:27.037316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:27.037316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:27.037316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:27.037316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:27.037316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.037316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:27.037316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:27.037316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.037316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:27.037316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:27.037316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.037316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:27.037316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:27.037316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:27.037596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:27.037896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:27.037896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:27.037896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:27.037896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:27.037896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:27.037896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.037896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:27.054196 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:27.054356 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:27.054435 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:27.054551 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:27.054841 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:27.055005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.055369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:27.055369 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:27.055369 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:27.055369 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:27.055369 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:27.055369 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:27.055369 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.055369 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:27.055911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.055911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:27.055911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:27.055911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:27.055911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.055911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:27.055911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:27.055911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:27.055911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.055911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:27.055911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:27.055911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:27.055911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:27.055911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:27.055911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:27.056155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:27.056155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:27.056155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:27.056155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:27.056155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:27.056155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:27.056155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:27.056155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.056155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:27.056468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.056468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:27.056468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:27.056468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:27.056468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.056468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:27.056468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:27.056468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:27.056468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:27.056468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.056468 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:27.056684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:27.056684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:27.056684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.056684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:27.056684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:27.056684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:27.056684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:27.056684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:27.056684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:27.056684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:27.056684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:27.056684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:27.056684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:27.056684 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:27.056907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:27.056907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:27.056907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.056907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.057195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:27.057480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:27.057824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:27.057824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:27.057824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.057824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:27.289273 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:27.289492 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:27.289638 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:27.289777 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:27.290098 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:27.290279 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:27.290400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.290827 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:27.290827 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:27.290827 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:27.290827 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:27.290827 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:27.290827 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:27.290827 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.290827 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:27.291382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.291382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:27.291382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:27.291382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:27.291382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.291382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:27.291382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:27.291382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.291382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:27.291382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:27.291382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:27.291382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.291635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:27.291635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:27.291635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:27.291635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:27.291635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:27.291635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:27.291635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:27.291635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:27.291635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:27.291635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:27.291635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:27.291635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:27.291635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:27.291862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.291862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:27.292078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.292078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:27.292078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:27.292078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:27.292078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:27.292078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:27.292078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.292078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:27.292078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:27.292078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.292078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:27.292078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:27.292078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.292078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:27.292078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:27.292078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:27.292351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:27.292714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:27.292714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.292714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:27.301169 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:27.301345 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:27.301422 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:27.302025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:27.328034 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:27.328188 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.328629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:27.328629 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:27.328629 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:27.328629 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:27.328629 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:27.328629 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:27.328629 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.328629 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:27.329304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.329304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:27.329304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:27.329304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:27.329304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.329304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:27.329304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:27.329304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.329304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:27.329304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:27.329304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:27.329304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.329656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:27.329656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:27.329656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:27.329656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:27.329656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:27.329656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:27.329656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:27.329656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:27.329656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:27.329656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:27.329656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:27.329656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:27.329656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:27.329888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.329888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:27.329969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.329969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:27.329969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:27.329969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:27.329969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:27.329969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:27.329969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.329969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:27.329969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:27.329969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.329969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:27.329969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:27.329969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.329969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:27.329969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:27.329969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:27.330297 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:27.330654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:27.330654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:27.330654 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:31.978569 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:31.980058 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:31.980058 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:31.980586 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:31.980941 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DelStaticRoutesCommand(_result=None, lrouter=neutron-f0dd56dc-a2bd-4aa6-97b5-e3f20c5afd4e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:31.981247 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DelStaticRoutesCommand(_result=None, lrouter=neutron-2d75de64-5181-4fce-b9ab-27f76f9c272e, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:31.981505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:31.982558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:31.982558 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:31.982558 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:31.982558 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:31.982558 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:31.982558 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:31.982558 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:31.982558 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:31.983660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:31.983660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:31.983660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:31.983660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:31.983660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:31.983660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:31.983660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:31.983660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:31.983660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:31.983660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:31.983660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:31.983660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:31.985481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:31.985481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:31.985481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:31.985481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:31.985481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:31.985481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:31.985481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:31.985481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:31.985481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:31.985481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:31.985481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:31.985481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:31.985481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:31.985890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:31.985890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:31.985890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:31.985890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:31.985890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:31.985890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:31.985890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:31.985890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:31.985890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:31.985890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:31.985890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:31.985890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:31.985890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:31.985890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:31.985890 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:31.986289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:31.986703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:31.986703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:31.986703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:31.986703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:31.986703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:31.986703 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:32.034401 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:32.035212 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:32.035212 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:32.035212 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:51:32.035742 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:32.036069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.036628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:32.036628 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:32.036628 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:32.036628 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:32.036628 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:32.036628 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:32.036628 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.036628 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:32.037508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.037508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:32.037508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:32.037508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:32.037508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.037508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:32.037508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:32.037508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:32.037508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.037508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:32.037508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:32.037508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:32.037508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:32.037508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:32.037508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:32.038795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:32.038795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:32.038795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:32.038795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:32.038795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:32.038795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:32.038795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:32.038795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.038795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:32.038795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.038795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:32.038795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:32.038795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:32.038795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.039127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:32.039127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:32.039127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:51:32.039127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:32.039127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:32.039127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.039127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:32.039127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:32.039127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:32.039127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.039127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:32.039127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:32.039127 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:32.039525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:32.039525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:32.039525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:32.039525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:32.039525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:32.039525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:32.039525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:32.039525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:32.039525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:32.039525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:32.039525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.039525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:32.041519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.041519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:32.041519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:32.041519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:32.041519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:32.041519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:32.041519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.041519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:32.041519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:32.041519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.041519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:32.041519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:32.041519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.041519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:32.041519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:32.041519 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:32.042889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:32.043822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:32.043822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:32.043822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:32.043822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:32.043822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:32.043822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.043822 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:32.054622 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:32.054846 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:32.054935 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:32.055037 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:32.055355 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:32.055614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.056086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:32.056086 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:32.056086 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:32.056086 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:32.056086 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:32.056086 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:32.056086 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.056086 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:32.056748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.056748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:32.056748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:32.056748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:32.056748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.056748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:32.056748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:32.056748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:32.056748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.056748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:32.056748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:32.056748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:32.056748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:32.056748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:32.056748 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:32.057029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:32.057029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:32.057029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:32.057029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:32.057029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:32.057029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:32.057029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:32.057029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.057029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:32.057656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.057656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:32.057656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:32.057656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:32.057656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.057656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:32.057656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:32.057656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:32.057656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:32.057656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.057656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:32.057942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:32.057942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:32.057942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.057942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:32.057942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:32.057942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:32.057942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:32.057942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:32.057942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:32.057942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:32.057942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:32.057942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:32.057942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:32.057942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:32.058159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:32.058159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:32.058159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.058159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.058852 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:32.059183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:32.059487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:32.059487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:32.059487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.059487 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:32.289739 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:32.290045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:32.290189 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:32.290379 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:32.290734 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:32.291007 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:32.291213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.291710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:32.291710 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:32.291710 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:32.291710 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:32.291710 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:32.291710 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:32.291710 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.291710 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:32.292402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.292402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:32.292402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:32.292402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:32.292402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.292402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:32.292402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:32.292402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.292402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:32.292402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:32.292402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:32.292402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.292672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:32.292672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:32.292672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:32.292672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:32.292672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:32.292672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:32.292672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:32.292672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:32.292672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:32.292672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:32.292672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:32.292672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:32.292672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:32.292888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.292888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:32.293523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.293523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:32.293523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:32.293523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:32.293523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:32.293523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:32.293523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.293523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:32.293523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:32.293523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.293523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:32.293523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:32.293523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.293523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:32.293523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:32.293523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:32.293835 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:32.294135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:32.294135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.294135 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:32.301529 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:32.301778 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:32.301897 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:32.302496 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:32.328880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:32.329012 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.329486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:32.329486 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:32.329486 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:32.329486 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:32.329486 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:32.329486 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:32.329486 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.329486 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:32.330222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.028 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.330222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:32.330222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:32.330222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:32.330222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.330222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:32.330222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:32.330222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.330222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:32.330222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:32.330222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:32.330222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.330503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:32.330503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:32.330503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:32.330503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:32.330503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:32.330503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:32.330503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:32.330503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:32.330503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:32.330503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:32.330503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:32.330503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:32.330503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:32.330770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.330770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:32.331309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.331309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:32.331309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:32.331309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:32.331309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:32.331309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:32.331309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.331309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:32.331309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:32.331309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.331309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:32.331309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:32.331309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.331309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:32.331309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:32.331309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:32.331696 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:32.332004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:32.332004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:32.332004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:36.979513 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:36.979909 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:36.979909 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:36.980351 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:36.980490 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:36.980947 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:36.980947 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:36.980947 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:36.980947 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:36.980947 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:36.980947 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:36.980947 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:36.980947 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:36.981616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:36.981616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:36.981616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:36.981616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:36.981616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:36.981616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:36.981616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:36.981616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:36.981616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:36.981616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:36.981616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:36.981616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:36.981979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:36.981979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:36.981979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:36.981979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:36.981979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:36.981979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:36.981979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:36.981979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:36.981979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:36.981979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:36.981979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:36.981979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:36.981979 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:36.982338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:36.982338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:36.982338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:36.982338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:36.982338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:36.982338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:36.982338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:36.982338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:36.982338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:36.982338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:36.982338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:36.982338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:36.982338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:36.982338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:36.982338 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:36.982784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:36.983305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:36.983305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:36.983305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:36.983305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:36.983305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:36.983305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:37.035064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:37.035500 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:37.035626 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:37.035750 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:51:37.036244 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:37.036402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.036821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:37.036821 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:37.036821 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:37.036821 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:37.036821 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:37.036821 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:37.036821 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.036821 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:37.037406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.037406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:37.037406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:37.037406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:37.037406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.037406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:37.037406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:37.037406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:37.037406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.037406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:37.037406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:37.037406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:37.037406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:37.037406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:37.037406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:37.037659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:37.037659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:37.037659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:37.037659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:37.037659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:37.037659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:37.037659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:37.037659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.037659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:37.038014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.038014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:37.038014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:37.038014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:37.038014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.038014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:37.038014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:37.038014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:51:37.038014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:37.038014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:37.038014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.038014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:37.038209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:37.038209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:37.038209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.038209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:37.038209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:37.038209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:37.038209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:37.038209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:37.038209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:37.038209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:37.038209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:37.038209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:37.038209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:37.038209 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:37.038420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:37.038420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:37.038420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.038420 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:37.038742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.038742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:37.038742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:37.038742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:37.038742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:37.038742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:37.038742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.038742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:37.038742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:37.038742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.038742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:37.038742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:37.038742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.038742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:37.038742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:37.038742 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:37.039006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:37.039279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:37.039279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:37.039279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:37.039279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:37.039279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:37.039279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.039279 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:37.054963 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:37.055269 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:37.055428 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:37.055589 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:37.055966 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:37.056160 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.056578 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:37.056578 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:37.056578 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:37.056578 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:37.056578 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:37.056578 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:37.056578 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.056578 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:37.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:37.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:37.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:37.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:37.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:37.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:37.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:37.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:37.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:37.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:37.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:37.057270 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:37.057522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:37.057522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:37.057522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:37.057522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:37.057522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:37.057522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:37.057522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:37.057522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.057522 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:37.057901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.057901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:37.057901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:37.057901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:37.057901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.057901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:37.057901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:37.057901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:37.057901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:37.057901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.057901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:37.058114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:37.058114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:37.058114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.058114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:37.058114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:37.058114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:37.058114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:37.058114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:37.058114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:37.058114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:37.058114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:37.058114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:37.058114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:37.058114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:37.058357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:37.058357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:37.058357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.058357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.058702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:37.058996 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:37.059291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:37.059291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:37.059291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.059291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:37.290102 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:37.290868 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:37.290946 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:37.291105 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:37.292888 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:37.293056 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:37.293190 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.293709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:37.293709 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:37.293709 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:37.293709 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:37.293709 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:37.293709 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:37.293709 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.293709 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:37.294407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.003 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.294407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:37.294407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:37.294407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:37.294407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.294407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:37.294407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:37.294407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.294407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:37.294407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:37.294407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:37.294407 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.294811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:37.294811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:37.294811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:37.294811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:37.294811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:37.294811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:37.294811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:37.294811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:37.294811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:37.294811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:37.294811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:37.294811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:37.294811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:37.295164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.295164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:37.295164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.295164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:37.295164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:37.295164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:37.295164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:37.295164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:37.295164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.295164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:37.295164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:37.295164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.295164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:37.295164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:37.295164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:37.295446 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:37.295927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:37.295927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:37.295927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:37.295927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:37.295927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.295927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:37.301907 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:37.302158 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:37.302243 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:37.302904 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:37.331861 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:37.332874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.333409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:37.333409 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:37.333409 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:37.333409 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:37.333409 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:37.333409 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:37.333409 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.333409 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:37.334134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.334134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:37.334134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:37.334134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:37.334134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.334134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:37.334134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:37.334134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.334134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:37.334134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:37.334134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:37.334134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.334511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:37.334511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:37.334511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:37.334511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:37.334511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:37.334511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:37.334511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:37.334511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:37.334511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:37.334511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:37.334511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:37.334511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:37.334511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:37.334910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.334910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:37.334910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.334910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:37.334910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:37.334910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:37.334910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:37.334910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:37.334910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.334910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:37.334910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:37.334910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.334910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:37.334910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:37.334910 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:37.335331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:37.335872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:37.335872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:37.335872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:37.335872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:37.335872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:37.335872 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:41.980035 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:41.980528 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:41.980528 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:41.981186 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:41.981395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:41.981929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:41.981929 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:41.981929 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:41.981929 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:41.981929 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:41.981929 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:41.981929 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:41.981929 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:41.982648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:41.982648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:41.982648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:41.982648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:41.982648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:41.982648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:41.982648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:41.982648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:41.982648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:41.982648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:41.982648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:41.982648 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:41.982930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:41.982930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:41.982930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:41.982930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:41.982930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:41.982930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:41.982930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:41.982930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:41.982930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:41.982930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:41.982930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:41.982930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:41.982930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:41.983170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:41.983170 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:41.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:41.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:41.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:41.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:41.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:41.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:41.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:41.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:41.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:41.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:41.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:41.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:41.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:41.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:41.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:41.983337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:41.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:41.984007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:41.984007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:41.984007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:42.035722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:42.035936 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:42.036058 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:42.036118 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:51:42.036619 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:42.036833 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.037294 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:42.037294 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:42.037294 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:42.037294 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:42.037294 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:42.037294 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:42.037294 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.037294 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:42.037951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.037951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:42.037951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:42.037951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:42.037951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.037951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:42.037951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:42.037951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:42.037951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.037951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:42.037951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:42.037951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:42.037951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:42.037951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:42.037951 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:42.038402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:42.038402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:42.038402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:42.038402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:42.038402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:42.038402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:42.038402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:42.038402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.038402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:42.038626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.038626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:42.038626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:42.038626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:42.038626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.038626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:42.038626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:42.038626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:51:42.038626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:42.038626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:42.038626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.038626 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:42.038911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:42.038911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:42.038911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.038911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:42.038911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:42.038911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:42.038911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:42.038911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:42.038911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:42.038911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:42.038911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:42.038911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:42.038911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:42.038911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:42.039152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:42.039152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:42.039152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.039152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:42.039333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.039333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:42.039333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:42.039333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:42.039333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:42.039333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:42.039333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.039333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:42.039333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:42.039333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.039333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:42.039333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:42.039333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.039333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:42.039333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:42.039333 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:42.039606 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:42.039902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:42.039902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:42.039902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:42.039902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:42.039902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:42.039902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.039902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:42.055393 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:42.055700 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:42.055847 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:42.056194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:42.056749 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:42.056997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.057497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:42.057497 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:42.057497 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:42.057497 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:42.057497 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:42.057497 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:42.057497 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.057497 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:42.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:42.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:42.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:42.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:42.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:42.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:42.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:42.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:42.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:42.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:42.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:42.058315 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:42.058580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:42.058580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:42.058580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:42.058580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:42.058580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:42.058580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:42.058580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:42.058580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.058580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:42.059195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.059195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:42.059195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:42.059195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:42.059195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.059195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:42.059195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:42.059195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:42.059195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:42.059195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.059195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:42.059397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:42.059397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:42.059397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.059397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:42.059397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:42.059397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:42.059397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:42.059397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:42.059397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:42.059397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:42.059397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:42.059397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:42.059397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:42.059397 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:42.059619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:42.059619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:42.059619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.059619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.060417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:42.060743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:42.061110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:42.061110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:42.061110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.061110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:42.291701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:42.291701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:42.291701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:42.291701 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:42.293052 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:42.293052 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:42.293052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.293052 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:42.293052 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:42.293052 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:42.293052 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:42.293052 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:42.293052 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:42.293052 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.293052 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:42.293754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.293754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:42.293754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:42.293754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:42.293754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.293754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:42.293754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:42.293754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.293754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:42.293754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:42.293754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:42.293754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.294175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:42.294175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:42.294175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:42.294175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:42.294175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:42.294175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:42.294175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:42.294175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:42.294175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:42.294175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:42.294175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:42.294175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:42.294175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:42.294368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.294368 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:42.294620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.294620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:42.294620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:42.294620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:42.294620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:42.294620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:42.294620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.294620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:42.294620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:42.294620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.294620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:42.294620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:42.294620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.294620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:42.294620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:42.294620 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:42.295144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:42.295471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:42.295471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.295471 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:42.302227 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:42.302506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:42.302506 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:42.303140 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:42.330452 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:42.330646 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.331090 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:42.331090 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:42.331090 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:42.331090 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:42.331090 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:42.331090 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:42.331090 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.331090 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:42.331814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.331814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:42.331814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:42.331814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:42.331814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.331814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:42.331814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:42.331814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.331814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:42.331814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:42.331814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:42.331814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.332165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:42.332165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:42.332165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:42.332165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:42.332165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:42.332165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:42.332165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:42.332165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:42.332165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:42.332165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:42.332165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:42.332165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:42.332165 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:42.332480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.332480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:42.332565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.332565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:42.332565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:42.332565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:42.332565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:42.332565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:42.332565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.332565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:42.332565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:42.332565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.332565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:42.332565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:42.332565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.332565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:42.332565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:42.332565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:42.332971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:42.333705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:42.333705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:42.333705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:46.980427 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:46.981012 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:46.981069 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:46.981561 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:46.981743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:46.982244 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:46.982244 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:46.982244 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:46.982244 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:46.982244 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:46.982244 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:46.982244 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:46.982244 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:46.983067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:46.983067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:46.983067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:46.983067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:46.983067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:46.983067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:46.983067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:46.983067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:46.983067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:46.983067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:46.983067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:46.983067 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:46.983461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:46.983896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:46.983896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:46.984179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:46.984179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:46.984179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:46.984179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:46.984179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:46.984179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:46.984179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:46.984179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:46.984179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:46.984179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:46.984179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:46.984179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:46.984179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:46.984179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:46.984179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:46.984179 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:46.984503 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:46.984876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:46.984876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:46.984876 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:47.036078 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:47.036440 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:47.036548 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:47.036682 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:51:47.037106 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:47.037198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.037607 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:47.037607 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:47.037607 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:47.037607 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:47.037607 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:47.037607 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:47.037607 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.037607 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:47.038163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.038163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:47.038163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:47.038163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:47.038163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.038163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:47.038163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:47.038163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:47.038163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.038163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:47.038163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:47.038163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:47.038163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:47.038163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:47.038163 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:47.038631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:47.038631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:47.038631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:47.038631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:47.038631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:47.038631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:47.038631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:47.038631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.038631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:47.038914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.038914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:47.038914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:47.038914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:47.038914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.038914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:47.038914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:47.038914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:51:47.038914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:47.038914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:47.038914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.038914 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:47.039261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:47.039261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:47.039261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.039261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:47.039261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:47.039261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:47.039261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:47.039261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:47.039261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:47.039261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:47.039261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:47.039261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:47.039261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:47.039261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:47.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:47.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:47.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:47.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:47.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:47.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:47.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:47.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:47.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:47.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:47.040064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:47.040460 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:47.040994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:47.040994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:47.040994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:47.040994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:47.040994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:47.040994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:47.040994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:47.040994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:47.040994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:47.040994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:47.040994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.040994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:47.055844 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:47.056219 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:47.056329 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:47.056433 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:47.056729 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:47.056935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.057314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:47.057314 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:47.057314 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:47.057314 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:47.057314 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:47.057314 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:47.057314 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.057314 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:47.057871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.057871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:47.057871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:47.057871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:47.057871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.057871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:47.057871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:47.057871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:47.057871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.057871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:47.057871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:47.057871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:47.057871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:47.057871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:47.057871 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:47.058343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:47.058343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:47.058343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:47.058343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:47.058343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:47.058343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:47.058343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:47.058343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.058343 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:47.058602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.058602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:47.058602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:47.058602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:47.058602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.058602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:47.058602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:47.058602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:47.058602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:47.058602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.058602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:47.058923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:47.058923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:47.058923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.058923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:47.058923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:47.058923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:47.058923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:47.058923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:47.058923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:47.058923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:47.058923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:47.058923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:47.058923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:47.058923 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:47.059328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:47.059328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:47.059328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.059328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:47.059328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.059328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:47.059328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:47.059328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:47.059328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:47.059328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:47.059328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.059328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:47.059328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:47.059328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:47.059710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:47.060264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:47.060264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:47.060264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:47.060264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:47.060264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:47.060264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:47.060264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:47.060264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:47.060264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:47.060264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.060264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:47.291472 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:47.291892 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:47.292032 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:47.292180 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:47.292457 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:47.292667 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:47.292770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.293196 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:47.293196 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:47.293196 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:47.293196 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:47.293196 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:47.293196 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:47.293196 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.293196 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:47.293875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.293875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:47.293875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:47.293875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:47.293875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.293875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:47.293875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:47.293875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.293875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:47.293875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:47.293875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:47.293875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.294162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:47.294162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:47.294162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:47.294162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:47.294162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:47.294162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:47.294162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:47.294162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:47.294162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:47.294162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:47.294162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:47.294162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:47.294162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:47.294369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.294369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:47.294524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.294524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:47.294524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:47.294524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:47.294524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:47.294524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:47.294524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.294524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:47.294524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:47.294524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.294524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:47.294524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:47.294524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.294524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:47.294524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:47.294524 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:47.294841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:47.295147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:47.295147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.295147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:47.302517 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:47.302838 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:47.302972 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:47.303608 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:47.328831 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:47.329002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.329426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:47.329426 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:47.329426 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:47.329426 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:47.329426 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:47.329426 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:47.329426 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.329426 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:47.330108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.330108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:47.330108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:47.330108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:47.330108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.330108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:47.330108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:47.330108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.330108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:47.330108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:47.330108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:47.330108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.330411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:47.330411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:47.330411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:47.330411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:47.330411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:47.330411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:47.330411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:47.330411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:47.330411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:47.330411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:47.330411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:47.330411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:47.330411 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:47.330635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.330635 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:47.330916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.330916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:47.330916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:47.330916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:47.330916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:47.330916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:47.330916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.330916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:47.330916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:47.330916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.330916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:47.330916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:47.330916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.330916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:47.330916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:47.330916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:47.331206 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:47.331523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:47.331523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:47.331523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:51.980946 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:51.981429 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:51.981429 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:51.981724 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:51.981896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:51.982395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:51.982395 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:51.982395 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:51.982395 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:51.982395 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:51.982395 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:51.982395 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:51.982395 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:51.983264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:51.983264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:51.983264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:51.983264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:51.983264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:51.983264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:51.983264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:51.983264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:51.983264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:51.983264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:51.983264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:51.983264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:51.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:51.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:51.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:51.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:51.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:51.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:51.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:51.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:51.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:51.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:51.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:51.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:51.983644 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:51.983976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:51.983976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:51.984310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:51.984310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:51.984310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:51.984310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:51.984310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:51.984310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:51.984310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:51.984310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:51.984310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:51.984310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:51.984310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:51.984310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:51.984310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:51.984310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:51.984310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:51.984310 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:51.984754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:51.985139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:51.985139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:51.985139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:52.036501 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:52.036740 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:52.036843 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:52.036939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:51:52.037215 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:52.037371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.037784 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:52.037784 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:52.037784 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:52.037784 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:52.037784 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:52.037784 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:52.037784 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.037784 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:52.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:52.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:52.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:52.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:52.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:52.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:52.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:52.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:52.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:52.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:52.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:52.038296 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:52.038732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:52.038732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:52.038732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:52.038732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:52.038732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:52.038732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:52.038732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:52.038732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.038732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:52.039020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.039020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:52.039020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:52.039020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:52.039020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.039020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:52.039020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:52.039020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:51:52.039020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:52.039020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:52.039020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.039020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:52.039349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:52.039349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:52.039349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.039349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:52.039349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:52.039349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:52.039349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:52.039349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:52.039349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:52.039349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:52.039349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:52.039349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:52.039349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:52.039349 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:52.039701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:52.039701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:52.039701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.039701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:52.039701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.039701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:52.039701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:52.039701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:52.039701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:52.039701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:52.039701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.039701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:52.039701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:52.039701 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:52.040073 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:52.040520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:52.040520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:52.040520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:52.040520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:52.040520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:52.040520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:52.040520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:52.040520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:52.040520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:52.040520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:52.040520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.040520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:52.056183 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:52.056372 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:52.056428 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:52.056608 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:52.057033 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:52.057269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.057757 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:52.057757 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:52.057757 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:52.057757 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:52.057757 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:52.057757 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:52.057757 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.057757 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:52.058290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.058290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:52.058290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:52.058290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:52.058290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.058290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:52.058290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:52.058290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:52.058290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.058290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:52.058290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:52.058290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:52.058290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:52.058290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:52.058290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:52.058738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:52.058738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:52.058738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:52.058738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:52.058738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:52.058738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:52.058738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:52.058738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.058738 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:52.059005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.059005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:52.059005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:52.059005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:52.059005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.059005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:52.059005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:52.059005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:52.059005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:52.059005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.059005 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:52.059320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:52.059320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:52.059320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.059320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:52.059320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:52.059320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:52.059320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:52.059320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:52.059320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:52.059320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:52.059320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:52.059320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:52.059320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:52.059320 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:52.059700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:52.059700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:52.059700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.059700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:52.059700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.059700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:52.059700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:52.059700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:52.059700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:52.059700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:52.059700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.059700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:52.059700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:52.059700 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:52.060113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:52.060657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:52.060657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:52.060657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:52.060657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:52.060657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:52.060657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:52.060657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:52.060657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:52.060657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:52.060657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.060657 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:52.291939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:52.292156 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:52.292255 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:52.292393 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:52.292858 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:52.293057 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:52.293169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.293615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:52.293615 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:52.293615 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:52.293615 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:52.293615 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:52.293615 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:52.293615 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.293615 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:52.294304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.294304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:52.294304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:52.294304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:52.294304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.294304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:52.294304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:52.294304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.294304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:52.294304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:52.294304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:52.294304 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.294521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:52.294521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:52.294521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:52.294521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:52.294521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:52.294521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:52.294521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:52.294521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:52.294521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:52.294521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:52.294521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:52.294521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:52.294521 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:52.294743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.294743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:52.294975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.294975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:52.294975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:52.294975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:52.294975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:52.294975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:52.294975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.294975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:52.294975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:52.294975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.294975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:52.294975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:52.294975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.294975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:52.294975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:52.294975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:52.295239 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:52.295564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:52.295564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.295564 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:52.302845 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:52.303028 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:52.303119 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:52.303712 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:52.329647 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:52.329925 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.330497 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:52.330497 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:52.330497 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:52.330497 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:52.330497 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:52.330497 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:52.330497 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.330497 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:52.331286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.331286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:52.331286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:52.331286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:52.331286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.331286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:52.331286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:52.331286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.331286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:52.331286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:52.331286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:52.331286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.331691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:52.331691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:52.331691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:52.331691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:52.331691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:52.331691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:52.331691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:52.331691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:52.331691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:52.331691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:52.331691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:52.331691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:52.331691 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:52.332027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.332027 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:52.332415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.332415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:52.332415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:52.332415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:52.332415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:52.332415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:52.332415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.332415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:52.332415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:52.332415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.332415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:52.332415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:52.332415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.332415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:52.332415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:52.332415 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:52.332888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:52.333412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:52.333412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:52.333412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:56.981282 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:56.981725 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:56.981913 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:56.982602 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:56.982723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:56.983091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:56.983091 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:56.983091 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:56.983091 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:56.983091 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:56.983091 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:56.983091 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:56.983091 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:56.983877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:56.983877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:56.983877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:56.983877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:56.983877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:56.983877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:56.983877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:56.983877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:56.983877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:56.983877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:56.983877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:56.983877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:56.984111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:56.984111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:56.984111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:56.984111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:56.984111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:56.984111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:56.984111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:56.984111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:56.984111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:56.984111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:56.984111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:56.984111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:56.984111 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:56.984357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:56.984357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:56.984615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:56.984615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:56.984615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:56.984615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:56.984615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:56.984615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:56.984615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:56.984615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:56.984615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:56.984615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:56.984615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:56.984615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:56.984615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:56.984615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:56.984615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:56.984615 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:56.984931 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:56.985238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:56.985238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:56.985238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:57.036901 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:57.037279 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:57.037386 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:57.037484 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:51:57.037932 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:57.038102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.038516 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:57.038516 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:57.038516 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:57.038516 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:57.038516 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:57.038516 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:57.038516 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.038516 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:57.039139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.039139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:57.039139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:57.039139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:57.039139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.039139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:57.039139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:57.039139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:57.039139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.039139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:57.039139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:57.039139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:57.039139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:57.039139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:57.039139 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:57.039404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:57.039404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:57.039404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:57.039404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:57.039404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:57.039404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:57.039404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:57.039404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.039404 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:57.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:57.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:57.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:57.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:57.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:57.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:51:57.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:57.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:57.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.039723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:57.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:57.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:57.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:57.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:57.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:57.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:57.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:57.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:57.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:57.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:57.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:57.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:57.039971 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:57.040216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:57.040216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:57.040216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.040216 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:57.040462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.040462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:57.040462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:57.040462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:57.040462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:57.040462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:57.040462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.040462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:57.040462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:57.040462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.040462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:57.040462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:57.040462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.040462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:57.040462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:57.040462 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:57.040770 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:57.041063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:57.041063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:57.041063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:57.041063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:57.041063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:57.041063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.041063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:57.056507 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:57.056871 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:57.056952 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:57.057064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:57.057416 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:57.057641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.058115 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:57.058115 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:57.058115 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:57.058115 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:57.058115 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:57.058115 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:57.058115 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.058115 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:57.058717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.058717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:51:57.058717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:57.058717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:57.058717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.058717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:57.058717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:57.058717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:51:57.058717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.058717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:51:57.058717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:57.058717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:51:57.058717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:57.058717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:51:57.058717 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:51:57.059199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:57.059199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:51:57.059199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:57.059199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:51:57.059199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:51:57.059199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:57.059199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:51:57.059199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.059199 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:51:57.059469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.059469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:57.059469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:57.059469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:57.059469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.059469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:57.059469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:57.059469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:57.059469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:57.059469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.059469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:57.059811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:57.059811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:57.059811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.059811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:57.059811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:57.059811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:57.059811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:57.059811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:57.059811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:57.059811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:57.059811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:57.059811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:57.059811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:57.059811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:57.060205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:57.060205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:57.060205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.060205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:57.060205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.060205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:57.060205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:57.060205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:57.060205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:57.060205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:57.060205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.060205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:57.060205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:57.060205 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:57.060632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:57.061169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:57.061169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:57.061169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:57.061169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:57.061169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:57.061169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:57.061169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:57.061169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:57.061169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:57.061169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.061169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:57.292323 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:57.292732 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:57.292883 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:57.293073 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:57.293418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:57.293655 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:57.293773 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.294256 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:57.294256 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:57.294256 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:57.294256 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:57.294256 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:57.294256 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:57.294256 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.294256 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:57.295219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.295219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:57.295219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:57.295219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:57.295219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.295219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:57.295219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:57.295219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.295219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:57.295219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:57.295219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:57.295219 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.295436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:57.295436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:57.295436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:57.295436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:57.295436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:57.295436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:57.295436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:57.295436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:57.295436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:57.295436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:57.295436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:57.295436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:57.295436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:57.295653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.295653 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:57.296221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.296221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:57.296221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:57.296221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:57.296221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:57.296221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:57.296221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.296221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:57.296221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:57.296221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.296221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:57.296221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:57.296221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.296221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:57.296221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:57.296221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:57.296483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:57.296798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:57.296798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.296798 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:51:57.303132 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:51:57.303385 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:51:57.303502 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:51:57.304267 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:51:57.333930 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:51:57.334159 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.334710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:51:57.334710 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:57.334710 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:51:57.334710 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:51:57.334710 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:57.334710 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:51:57.334710 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.334710 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:51:57.335412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.031 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.335412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:51:57.335412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:57.335412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:51:57.335412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.335412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:57.335412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:57.335412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.335412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:57.335412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:57.335412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:51:57.335412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.335672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:51:57.335672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:57.335672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:51:57.335672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:57.335672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:51:57.335672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:51:57.335672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:57.335672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:51:57.335672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:57.335672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:51:57.335672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:51:57.335672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:57.335672 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:51:57.335893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.335893 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:51:57.336136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.336136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:51:57.336136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:51:57.336136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:51:57.336136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:51:57.336136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:51:57.336136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.336136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:51:57.336136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:51:57.336136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.336136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:51:57.336136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:51:57.336136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.336136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:51:57.336136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:51:57.336136 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:51:57.336381 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:51:57.336689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:51:57.336689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:51:57.336689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:01.981890 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:01.982616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:01.982616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:01.982824 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:01.983028 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:01.983493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:01.983493 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:01.983493 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:01.983493 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:01.983493 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:01.983493 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:01.983493 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:01.983493 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:01.984243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:01.984489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:01.984489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:01.984489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:01.984489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:01.984489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:01.984489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:01.984489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:01.984489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:01.984489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:01.984489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:01.984489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:01.984489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:01.984489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:01.984716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:01.984716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:01.985020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:01.985020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:01.985020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:01.985020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:01.985020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:01.985020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:01.985020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:01.985020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:01.985020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:01.985020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:01.985020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:01.985020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:01.985020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:01.985020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:01.985020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:01.985020 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:01.985298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:01.985640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:01.985640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:01.985640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:02.037358 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:02.037581 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:02.037645 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:02.037737 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:52:02.038229 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:02.038433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.038927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:02.038927 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:02.038927 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:02.038927 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:02.038927 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:02.038927 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:02.038927 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.038927 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:02.039510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.039510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:02.039510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:02.039510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:02.039510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.039510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:02.039510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:02.039510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:02.039510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.039510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:02.039510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:02.039510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:02.039510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:02.039510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:02.039510 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:02.039820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:02.039820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:02.039820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:02.039820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:02.039820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:02.039820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:02.039820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:02.039820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.039820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:02.040110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.040110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:02.040110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:02.040110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:02.040110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.040110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:02.040110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:02.040110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:52:02.040110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:02.040110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:02.040110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.040110 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:02.040533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:02.040533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:02.040533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.040533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:02.040533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:02.040533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:02.040533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:02.040533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:02.040533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:02.040533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:02.040533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:02.040533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:02.040533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:02.040533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:02.041102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:02.041102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:02.041102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.041102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:02.041102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.041102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:02.041102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:02.041102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:02.041102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:02.041102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:02.041102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.041102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:02.041102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:02.041102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:02.041398 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:02.041726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:02.041726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:02.041726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:02.041726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:02.041726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:02.041726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:02.041726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:02.041726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:02.041726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:02.041726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:02.041726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.041726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:02.056943 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:02.057112 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:02.057201 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:02.057306 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:02.057686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:02.057907 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.058305 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:02.058305 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:02.058305 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:02.058305 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:02.058305 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:02.058305 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:02.058305 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.058305 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:02.058963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.058963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:02.058963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:02.058963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:02.058963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.058963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:02.058963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:02.058963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:02.058963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.058963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:02.058963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:02.058963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:02.058963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:02.058963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:02.058963 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:02.059207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:02.059207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:02.059207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:02.059207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:02.059207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:02.059207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:02.059207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:02.059207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.059207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:02.059602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.059602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:02.059602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:02.059602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:02.059602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.059602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:02.059602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:02.059602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:02.059602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:02.059602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.059602 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:02.059824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:02.059824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:02.059824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.059824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:02.059824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:02.059824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:02.059824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:02.059824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:02.059824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:02.059824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:02.059824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:02.059824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:02.059824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:02.059824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:02.060019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:02.060019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:02.060019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.060019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.060306 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:02.060614 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:02.060918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:02.060918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:02.060918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.060918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:02.292750 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:02.293045 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:02.293145 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:02.293318 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:02.293879 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:02.294157 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:02.294302 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.294899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:02.294899 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:02.294899 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:02.294899 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:02.294899 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:02.294899 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:02.294899 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.294899 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:02.295754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.295754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:02.295754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:02.295754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:02.295754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.295754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:02.295754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:02.295754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.295754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:02.295754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:02.295754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:02.295754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.296113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:02.296113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:02.296113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:02.296113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:02.296113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:02.296113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:02.296113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:02.296113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:02.296113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:02.296113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:02.296113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:02.296113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:02.296113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:02.296413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.296413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:02.296884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.296884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:02.296884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:02.296884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:02.296884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:02.296884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:02.296884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.296884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:02.296884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:02.296884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.296884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:02.296884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:02.296884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.296884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:02.296884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:02.296884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:02.297168 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:02.297458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:02.297458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.297458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:02.303438 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:02.303690 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:02.303781 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:02.304437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:02.332263 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:02.332515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.333245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:02.333245 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:02.333245 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:02.333245 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:02.333245 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:02.333245 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:02.333245 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.333245 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:02.334183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.334183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:02.334183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:02.334183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:02.334183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.334183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:02.334183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:02.334183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.334183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:02.334183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:02.334183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:02.334183 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.334504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:02.334504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:02.334504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:02.334504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:02.334504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:02.334504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:02.334504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:02.334504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:02.334504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:02.334504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:02.334504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:02.334504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:02.334504 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:02.334743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.334743 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:02.335298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.335298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:02.335298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:02.335298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:02.335298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:02.335298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:02.335298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.335298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:02.335298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:02.335298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.335298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:02.335298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:02.335298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.335298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:02.335298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:02.335298 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:02.335600 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:02.336059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:02.336059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:02.336059 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:06.982351 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:06.982830 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:06.982830 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:06.983307 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:06.983502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:06.984155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:06.984155 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:06.984155 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:06.984155 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:06.984155 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:06.984155 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:06.984155 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:06.984155 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:06.984902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:06.984902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:06.984902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:06.984902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:06.984902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:06.984902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:06.984902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:06.984902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:06.984902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:06.984902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:06.984902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:06.984902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:06.985102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:06.985102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:06.985102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:06.985102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:06.985102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:06.985102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:06.985102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:06.985102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:06.985102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:06.985102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:06.985102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:06.985102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:06.985102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:06.985300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:06.985300 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:06.985612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:06.985612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:06.985612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:06.985612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:06.985612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:06.985612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:06.985612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:06.985612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:06.985612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:06.985612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:06.985612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:06.985612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:06.985612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:06.985612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:06.985612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:06.985612 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:06.985874 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:06.986152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:06.986152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:06.986152 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:07.037740 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:07.038061 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:07.038174 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:07.038290 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:52:07.038860 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:07.039086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.039681 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:07.039681 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:07.039681 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:07.039681 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:07.039681 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:07.039681 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:07.039681 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.039681 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:07.040347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.040347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:07.040347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:07.040347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:07.040347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.040347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:07.040347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:07.040347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:07.040347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.040347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:07.040347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:07.040347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:07.040347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:07.040347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:07.040347 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:07.040596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:07.040596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:07.040596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:07.040596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:07.040596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:07.040596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:07.040596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:07.040596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.040596 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:52:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.041037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:07.041243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:07.041243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:07.041243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.041243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:07.041243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:07.041243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:07.041243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:07.041243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:07.041243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:07.041243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:07.041243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:07.041243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:07.041243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:07.041243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:07.041476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:07.041476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:07.041476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.041476 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:07.041797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.041797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:07.041797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:07.041797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:07.041797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:07.041797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:07.041797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.041797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:07.041797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:07.041797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.041797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:07.041797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:07.041797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.041797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:07.041797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:07.041797 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:07.042096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:07.042403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:07.042403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:07.042403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:07.042403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:07.042403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:07.042403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.042403 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:07.057250 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:07.057553 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:07.057691 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:07.057833 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:07.058216 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:07.058472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.059066 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:07.059066 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:07.059066 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:07.059066 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:07.059066 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:07.059066 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:07.059066 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.059066 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:07.059714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.059714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:07.059714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:07.059714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:07.059714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.059714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:07.059714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:07.059714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:07.059714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.059714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:07.059714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:07.059714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:07.059714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:07.059714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:07.059714 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:07.060091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:07.060091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:07.060091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:07.060091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:07.060091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:07.060091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:07.060091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:07.060091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.060091 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:07.060894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.060894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:07.060894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:07.060894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:07.060894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.060894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:07.060894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:07.060894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:07.060894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:07.060894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.060894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:07.061074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:07.061074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:07.061074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.061074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:07.061074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:07.061074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:07.061074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:07.061074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:07.061074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:07.061074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:07.061074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:07.061074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:07.061074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:07.061074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:07.061318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:07.061318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:07.061318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.061318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:07.062006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:07.062280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:07.062280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:07.062280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.062280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:07.293238 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:07.293657 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:07.293836 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:07.294047 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:07.294569 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:07.294838 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:07.294942 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.295429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:07.295429 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:07.295429 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:07.295429 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:07.295429 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:07.295429 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:07.295429 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.295429 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:07.296414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.296414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:07.296414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:07.296414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:07.296414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.296414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:07.296414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:07.296414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.296414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:07.296414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:07.296414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:07.296414 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.296640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:07.296640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:07.296640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:07.296640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:07.296640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:07.296640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:07.296640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:07.296640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:07.296640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:07.296640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:07.296640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:07.296640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:07.296640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:07.296864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.296864 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:07.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:07.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:07.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:07.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:07.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:07.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:07.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:07.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:07.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:07.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:07.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:07.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:07.297812 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:07.298147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:07.298147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.298147 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:07.303806 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:07.304125 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:07.304269 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:07.305124 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:07.340759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:07.340759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.340759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:07.340759 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:07.340759 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:07.340759 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:07.340759 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:07.340759 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:07.340759 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.340759 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:07.340759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.340759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:07.340759 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:07.341069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:07.341069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.341069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:07.341069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:07.341069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.341069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:07.341069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:07.341069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:07.341069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.341069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:07.341069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:07.341069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:07.341069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:07.341069 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:07.341363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:07.341363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:07.341363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:07.341363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:07.341363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:07.341363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:07.341363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:07.341363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:07.341363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.341363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:07.341363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.341363 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:07.341580 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:07.341933 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:11.982758 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:11.983325 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:11.983325 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:11.983672 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:11.983904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:11.984356 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:11.984356 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:11.984356 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:11.984356 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:11.984356 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:11.984356 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:11.984356 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:11.984356 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:11.985212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:11.985212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:11.985212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:11.985212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:11.985212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:11.985212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:11.985212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:11.985212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:11.985212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:11.985212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:11.985212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:11.985212 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:11.985566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:11.985566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:11.985566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:11.985566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:11.985566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:11.985566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:11.985566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:11.985566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:11.985566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:11.985566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:11.985566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:11.985566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:11.985566 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:11.985919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:11.985919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:11.985919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:11.985919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:11.985919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:11.985919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:11.985919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:11.985919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:11.985919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:11.985919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:11.985919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:11.985919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:11.985919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:11.985919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:11.985919 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:11.986316 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:11.986837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:11.986837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:11.986837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:11.986837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:11.986837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:11.986837 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:12.038094 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:12.038470 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:12.038607 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:12.038710 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:52:12.039068 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:12.039245 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.039643 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:12.039643 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:12.039643 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:12.039643 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:12.039643 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:12.039643 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:12.039643 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.039643 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:12.040151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.040151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:12.040151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:12.040151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:12.040151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.040151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:12.040151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:12.040151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:12.040151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.040151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:12.040151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:12.040151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:12.040151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:12.040151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:12.040151 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:12.040426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:12.040426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:12.040426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:12.040426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:12.040426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:12.040426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:12.040426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:12.040426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.040426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:12.040699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.040699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:12.040699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:12.040699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:12.040699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.040699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:12.040699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:12.040699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:52:12.040699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:12.040699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:12.040699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.040699 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:12.040956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:12.040956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:12.040956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.040956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:12.040956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:12.040956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:12.040956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:12.040956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:12.040956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:12.040956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:12.040956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:12.040956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:12.040956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:12.040956 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:12.041214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:12.041214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:12.041214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.041214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:12.041383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.041383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:12.041383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:12.041383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:12.041383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:12.041383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:12.041383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.041383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:12.041383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:12.041383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.041383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:12.041383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:12.041383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.041383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:12.041383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:12.041383 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:12.041655 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:12.041981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:12.041981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:12.041981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:12.041981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:12.041981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:12.041981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.041981 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:12.057598 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:12.057954 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:12.058046 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:12.058170 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:12.058494 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:12.058723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.059144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:12.059144 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:12.059144 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:12.059144 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:12.059144 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:12.059144 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:12.059144 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.059144 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:12.059665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.059665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:12.059665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:12.059665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:12.059665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.059665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:12.059665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:12.059665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:12.059665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.059665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:12.059665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:12.059665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:12.059665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:12.059665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:12.059665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:12.059906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:12.059906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:12.059906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:12.059906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:12.059906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:12.059906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:12.059906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:12.059906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.059906 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:12.060247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.060247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:12.060247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:12.060247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:12.060247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.060247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:12.060247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:12.060247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:12.060247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:12.060247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.060247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:12.060472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:12.060472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:12.060472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.060472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:12.060472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:12.060472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:12.060472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:12.060472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:12.060472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:12.060472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:12.060472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:12.060472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:12.060472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:12.060472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:12.060695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:12.060695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:12.060695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.060695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.060966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:12.061277 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:12.061660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:12.061660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:12.061660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.061660 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:12.293704 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:12.294077 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:12.294154 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:12.294314 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:12.294625 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:12.294814 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:12.294904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.295293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:12.295293 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:12.295293 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:12.295293 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:12.295293 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:12.295293 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:12.295293 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.295293 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:12.295888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.295888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:12.295888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:12.295888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:12.295888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.295888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:12.295888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:12.295888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.295888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:12.295888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:12.295888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:12.295888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.296198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:12.296198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:12.296198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:12.296198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:12.296198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:12.296198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:12.296198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:12.296198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:12.296198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:12.296198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:12.296198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:12.296198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:12.296198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:12.296502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.296502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:12.296502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.296502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:12.296502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:12.296502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:12.296502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:12.296502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:12.296502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.296502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:12.296502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:12.296502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.296502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:12.296502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:12.296502 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:12.296901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:12.297341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:12.297341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:12.297341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:12.297341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:12.297341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.297341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:12.304183 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:12.304446 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:12.304530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:12.305276 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:12.329842 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:12.329997 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.330418 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:12.330418 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:12.330418 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:12.330418 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:12.330418 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:12.330418 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:12.330418 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.330418 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:12.331072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.331072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:12.331072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:12.331072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:12.331072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.331072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:12.331072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:12.331072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.331072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:12.331072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:12.331072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:12.331072 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.331394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:12.331394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:12.331394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:12.331394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:12.331394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:12.331394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:12.331394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:12.331394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:12.331394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:12.331394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:12.331394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:12.331394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:12.331394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:12.331725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.331725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:12.331725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.331725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:12.331725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:12.331725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:12.331725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:12.331725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:12.331725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.331725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:12.331725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:12.331725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.331725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:12.331725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:12.331725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:12.332093 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:12.332559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:12.332559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:12.332559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:12.332559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:12.332559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:12.332559 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:16.983221 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:16.983779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:16.983779 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:16.984851 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:16.984851 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:16.984937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:16.984937 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:16.984937 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:16.984937 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:16.984937 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:16.984937 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:16.984937 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:16.984937 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:16.985619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:16.985619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:16.985619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:16.985619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:16.985619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:16.985619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:16.985619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:16.985619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:16.985619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:16.985619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:16.985619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:16.985619 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:16.986036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:16.986036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:16.986036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:16.986036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:16.986036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:16.986036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:16.986036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:16.986036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:16.986036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:16.986036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:16.986036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:16.986036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:16.986036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:16.986234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:16.986234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:16.986436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:16.986436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:16.986436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:16.986436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:16.986436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:16.986436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:16.986436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:16.986436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:16.986436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:16.986436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:16.986436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:16.986436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:16.986436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:16.986436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:16.986436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:16.986436 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:16.987046 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:16.987473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:16.987473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:16.987473 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:17.038578 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:17.038978 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:17.039072 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:17.039159 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:52:17.040291 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:17.040291 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.040429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:17.040429 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:17.040429 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:17.040429 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:17.040429 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:17.040429 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:17.040429 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.040429 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:17.041062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.041062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:17.041062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:17.041062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:17.041062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.041062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:17.041062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:17.041062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:17.041062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.041062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:17.041062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:17.041062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:17.041062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:17.041062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:17.041062 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:17.041689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:17.041689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:17.041689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:17.041689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:17.041689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:17.041689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:17.041689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:17.041689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.041689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:17.041689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.041689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:17.041689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:17.041689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:17.041689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.042271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:17.042271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:17.042271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:52:17.042271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:17.042271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:17.042271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.042271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:17.042271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:17.042271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:17.042271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.042271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:17.042271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:17.042271 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:17.042755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:17.042755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:17.042755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:17.042755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:17.042755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:17.042755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:17.042755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:17.042755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:17.042755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:17.042755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:17.042755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.042755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:17.043038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.043038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:17.043038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:17.043038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:17.043038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:17.043038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:17.043038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.043038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:17.043038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:17.043038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.043038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:17.043038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:17.043038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.043038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:17.043038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:17.043038 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:17.043344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:17.043716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:17.043716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:17.043716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:17.043716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:17.043716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:17.043716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.043716 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:17.058113 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:17.058467 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:17.058579 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:17.058688 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:17.059795 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:17.059795 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.059926 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:17.059926 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:17.059926 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:17.059926 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:17.059926 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:17.059926 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:17.059926 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.059926 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:17.060523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.060523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:17.060523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:17.060523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:17.060523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.060523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:17.060523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:17.060523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:17.060523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.060523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:17.060523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:17.060523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:17.060523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:17.060523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:17.060523 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:17.060900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:17.060900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:17.060900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:17.060900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:17.060900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:17.060900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:17.060900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:17.060900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.060900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:17.061104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.061104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:17.061104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:17.061104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:17.061104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.061104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:17.061104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:17.061104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:17.061104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:17.061104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.061104 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:17.061366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:17.061366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:17.061366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.061366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:17.061366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:17.061366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:17.061366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:17.061366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:17.061366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:17.061366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:17.061366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:17.061366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:17.061366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:17.061366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:17.061689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:17.061689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:17.061689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.061689 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.061895 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:17.062274 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:17.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:17.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:17.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:17.294158 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:17.294549 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:17.294712 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:17.294885 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:17.295616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:17.295616 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:17.295702 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:17.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:17.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:17.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:17.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:17.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:17.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:17.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:17.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:17.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:17.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:17.297477 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:17.297862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.297862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:17.297862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:17.297862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:17.297862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.297862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:17.297862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:17.297862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:17.297862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:17.297862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:17.297862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:17.297862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:17.297862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:17.297862 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:17.298078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:17.298078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:17.298078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:17.298078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:17.298078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.298078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:17.298078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.298078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:17.298078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:17.298078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:17.298078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:17.298078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:17.298078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.298078 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:17.298303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:17.298631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:17.298631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:17.298631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:17.298631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:17.298631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:17.298631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:17.298631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:17.298631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:17.298631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.298631 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:17.304711 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:17.305034 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:17.305247 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:17.306160 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:17.333252 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:17.333496 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.335478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:17.335478 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:17.335478 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:17.335478 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:17.335478 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:17.335478 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:17.335478 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.335478 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:17.335478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.335478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:17.335478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:17.335478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:17.335478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.335478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:17.335478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:17.335811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.335811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:17.335811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:17.335811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:17.335811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.335811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:17.335811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:17.335811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:17.335811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:17.335811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:17.335811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:17.335811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:17.335811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:17.335811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:17.336026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:17.336026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:17.336026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:17.336026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:17.336026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.336026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:17.336026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.336026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:17.336026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:17.336026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:17.336026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:17.336026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:17.336026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.336026 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:17.336259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:17.336595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:17.336595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:17.336595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:17.336595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:17.336595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:17.336595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:17.336595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:17.336595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:17.336595 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:21.983694 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:21.985133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:21.985133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:21.985133 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:21.985133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:21.985133 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:21.985133 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:21.985133 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:21.985133 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:21.985133 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:21.985133 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:21.985133 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:21.985133 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:21.985726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:21.985726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:21.985726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:21.985726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:21.985726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:21.985726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:21.985726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:21.985726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:21.985726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:21.985726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:21.985726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:21.985726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:21.986024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:21.986024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:21.986024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:21.986024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:21.986024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:21.986024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:21.986024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:21.986024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:21.986024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:21.986024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:21.986024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:21.986024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:21.986024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:21.986318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:21.986318 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:21.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:21.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:21.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:21.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:21.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:21.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:21.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:21.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:21.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:21.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:21.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:21.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:21.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:21.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:21.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:21.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:21.987479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:21.988004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:21.988004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:21.988004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:22.039087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:22.039308 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:22.039396 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:22.039463 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:52:22.040264 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:22.040264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.040369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:22.040369 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:22.040369 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:22.040369 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:22.040369 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:22.040369 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:22.040369 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.040369 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:22.040950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.040950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:22.040950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:22.040950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:22.040950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.040950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:22.040950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:22.040950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:22.040950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.040950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:22.040950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:22.040950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:22.040950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:22.040950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:22.040950 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:22.041234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:22.041234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:22.041234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:22.041234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:22.041234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:22.041234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:22.041234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:22.041234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.041234 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:22.041533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.041533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:22.041533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:22.041533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:22.041533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.041533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:22.041533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:22.041533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:52:22.041533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:22.041533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:22.041533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.041533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:22.042064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:22.042064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:22.042064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.042064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:22.042064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:22.042064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:22.042064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:22.042064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:22.042064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:22.042064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:22.042064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:22.042064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:22.042064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:22.042064 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:22.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:22.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:22.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:22.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:22.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:22.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:22.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:22.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:22.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:22.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:22.042525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:22.042763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:22.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:22.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:22.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:22.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:22.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:22.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:22.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:22.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:22.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:22.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:22.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.043119 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:22.058562 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:22.058911 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:22.059065 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:22.059215 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:22.060095 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:22.060095 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.060251 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:22.060251 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:22.060251 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:22.060251 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:22.060251 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:22.060251 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:22.060251 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.060251 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:22.061009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.061009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:22.061009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:22.061009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:22.061009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.061009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:22.061009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:22.061009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:22.061009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.061009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:22.061009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:22.061009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:22.061009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:22.061009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:22.061009 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:22.061604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:22.061604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:22.061604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:22.061604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:22.061604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:22.061604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:22.061604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:22.061604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.061604 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:22.061922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.061922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:22.061922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:22.061922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:22.061922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.061922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:22.061922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:22.061922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:22.061922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:22.061922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.061922 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:22.062495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:22.062495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:22.062495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.062495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:22.062495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:22.062495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:22.062495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:22.062495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:22.062495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:22.062495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:22.062495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:22.062495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:22.062495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:22.062495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:22.062801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:22.062801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:22.062801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.062801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.062878 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:22.063173 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:22.063557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:22.063557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:22.063557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.063557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:22.294631 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:22.295050 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:22.295182 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:22.295341 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:22.296661 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:22.296661 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:22.296661 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.296821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:22.296821 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:22.296821 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:22.296821 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:22.296821 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:22.296821 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:22.296821 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.296821 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:22.297622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.297622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:22.297622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:22.297622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:22.297622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.297622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:22.297622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:22.297622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.297622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:22.297622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:22.297622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:22.297622 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.298008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:22.298008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:22.298008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:22.298008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:22.298008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:22.298008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:22.298008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:22.298008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:22.298008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:22.298008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:22.298008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:22.298008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:22.298008 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:22.298357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.298357 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:22.298429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.298429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:22.298429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:22.298429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:22.298429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:22.298429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:22.298429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.298429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:22.298429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:22.298429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.298429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:22.298429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:22.298429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.298429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:22.298429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:22.298429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:22.298857 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:22.299425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:22.299425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.299425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:22.305200 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:22.305463 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:22.305655 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:22.306504 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:22.311840 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG dbcounter [-] [89427] Writing DB stats neutron:SELECT=240 {{(pid=89427) stat_writer /opt/stack/data/venv/lib/python3.12/site-packages/dbcounter.py:115}} Jun 01 11:52:22.338636 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:22.339472 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.339670 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:22.339670 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:22.339670 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:22.339670 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:22.339670 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:22.339670 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:22.339670 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.339670 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:22.340719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.034 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.340719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:22.340719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:22.340719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:22.340719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.340719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:22.340719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:22.340719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.340719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:22.340719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:22.340719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:22.340719 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.341108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:22.341108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:22.341108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:22.341108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:22.341108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:22.341108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:22.341108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:22.341108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:22.341108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:22.341108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:22.341108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:22.341108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:22.341108 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:22.341422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.341422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:22.341763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.341763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:22.341763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:22.341763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:22.341763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:22.341763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:22.341763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.341763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:22.341763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:22.341763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.341763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:22.341763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:22.341763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.341763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:22.341763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:22.341763 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:22.342259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:22.342807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:22.342807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:22.342807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:26.984178 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:26.984676 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:26.984676 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:26.985128 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:26.985325 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:26.985806 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:26.985806 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:26.985806 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:26.985806 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:26.985806 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:26.985806 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:26.985806 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:26.985806 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:26.986493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:26.986493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:26.986493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:26.986493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:26.986493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:26.986493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:26.986493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:26.986493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:26.986493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:26.986493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:26.986493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:26.986493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:26.986961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:26.986961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:26.986961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:26.986961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:26.986961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:26.986961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:26.986961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:26.986961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:26.986961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:26.986961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:26.986961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:26.986961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:26.986961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:26.987198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:26.987198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:26.987236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:26.987236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:26.987236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:26.987236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:26.987236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:26.987236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:26.987236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:26.987236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:26.987236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:26.987236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:26.987236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:26.987236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:26.987236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:26.987236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:26.987236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:26.987236 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:26.987461 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:26.987802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:26.987802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:26.987802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:27.039437 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:27.039720 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:27.039839 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:27.039934 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:52:27.040632 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:27.040888 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.041443 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:27.041443 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:27.041443 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:27.041443 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:27.041443 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:27.041443 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:27.041443 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.041443 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:27.042071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.042071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:27.042071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:27.042071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:27.042071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.042071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:27.042071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:27.042071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:27.042071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.042071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:27.042071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:27.042071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:27.042071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:27.042071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:27.042071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:27.042454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:27.042454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:27.042454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:27.042454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:27.042454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:27.042454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:27.042454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:27.042454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.042454 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:27.042711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.042711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:27.042711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:27.042711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:27.042711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.042711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:27.042711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:27.042711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:52:27.042711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:27.042711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:27.042711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.042711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:27.043114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:27.043114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:27.043114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.043114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:27.043114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:27.043114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:27.043114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:27.043114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:27.043114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:27.043114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:27.043114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:27.043114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:27.043114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:27.043114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:27.043402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:27.043402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:27.043402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.043402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:27.043402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.043402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:27.043402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:27.043402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:27.043402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:27.043402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:27.043402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.043402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:27.043402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:27.043402 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:27.043628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:27.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:27.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:27.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:27.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:27.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:27.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:27.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:27.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:27.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:27.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:27.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.044013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:27.058880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:27.059087 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:27.059174 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:27.059272 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:27.059660 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:27.059904 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.060399 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:27.060399 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:27.060399 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:27.060399 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:27.060399 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:27.060399 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:27.060399 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.060399 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:27.061056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.061056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:27.061056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:27.061056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:27.061056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.061056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:27.061056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:27.061056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:27.061056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.061056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:27.061056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:27.061056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:27.061056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:27.061056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:27.061056 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:27.061481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:27.061481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:27.061481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:27.061481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:27.061481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:27.061481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:27.061481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:27.061481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.061481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:27.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:27.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:27.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:27.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:27.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:27.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:27.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:27.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.061656 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:27.061836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:27.061836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:27.061836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.061836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:27.061836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:27.061836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:27.061836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:27.061836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:27.061836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:27.061836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:27.061836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:27.061836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:27.061836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:27.061836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:27.062083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:27.062083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:27.062083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.062083 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.062417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:27.063063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:27.063431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:27.063431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:27.063431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.063431 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:27.295075 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:27.295406 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:27.295510 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:27.295733 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:27.296172 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:27.296385 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:27.296486 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.296953 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:27.296953 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:27.296953 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:27.296953 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:27.296953 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:27.296953 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:27.296953 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.296953 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:27.297794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.297794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:27.297794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:27.297794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:27.297794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.297794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:27.297794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:27.297794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.297794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:27.297794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:27.297794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:27.297794 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.298112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:27.298112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:27.298112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:27.298112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:27.298112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:27.298112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:27.298112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:27.298112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:27.298112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:27.298112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:27.298112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:27.298112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:27.298112 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:27.298425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.298425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:27.298896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.298896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:27.298896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:27.298896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:27.298896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:27.298896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:27.298896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.298896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:27.298896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:27.298896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.298896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:27.298896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:27.298896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.298896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:27.298896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:27.298896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:27.299194 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:27.299480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:27.299480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.299480 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:27.305516 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:27.305754 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:27.305865 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:27.306535 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:27.330924 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:27.331167 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.331802 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:27.331802 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:27.331802 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:27.331802 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:27.331802 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:27.331802 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:27.331802 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.331802 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:27.332489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.332489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:27.332489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:27.332489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:27.332489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.332489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:27.332489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:27.332489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.332489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:27.332489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:27.332489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:27.332489 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.332765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:27.332765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:27.332765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:27.332765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:27.332765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:27.332765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:27.332765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:27.332765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:27.332765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:27.332765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:27.332765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:27.332765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:27.332765 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:27.333141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.333141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:27.333242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.333242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:27.333242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:27.333242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:27.333242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:27.333242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:27.333242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.333242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:27.333242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:27.333242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.333242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:27.333242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:27.333242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.333242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:27.333242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:27.333242 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:27.333720 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:27.334264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:27.334264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:27.334264 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:31.984696 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:31.985247 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:31.985247 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:31.986276 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:31.986276 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:31.986366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:31.986366 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:31.986366 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:31.986366 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:31.986366 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:31.986366 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:31.986366 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:31.986366 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:31.987709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:31.987709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:31.987709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:31.987709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:31.987709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:31.987709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:31.987709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:31.987709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:31.987709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:31.987709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:31.987709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:31.987709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:31.987937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:31.987937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:31.987937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:31.987937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:31.987937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:31.987937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:31.987937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:31.987937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:31.987937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:31.987937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:31.987937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:31.987937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:31.987937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:31.988148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:31.988148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:31.988148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:31.988148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:31.988148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:31.988148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:31.988148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:31.988148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:31.988148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:31.988148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:31.988148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:31.988148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:31.988148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:31.988148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:31.988148 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:31.988458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:31.988756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:31.988756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:31.988756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:31.988756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:31.988756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:31.988756 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:32.039868 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:32.040464 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:32.040582 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:32.040705 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:52:32.041025 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:32.041214 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:32.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:32.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:32.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:32.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:32.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:32.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.041649 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:32.042257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.042257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:32.042257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:32.042257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:32.042257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.042257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:32.042257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:32.042257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:32.042257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.042257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:32.042257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:32.042257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:32.042257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:32.042257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:32.042257 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:32.042609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:32.042609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:32.042609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:32.042609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:32.042609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:32.042609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:32.042609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:32.042609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.042609 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:32.043057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.043057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:32.043057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:32.043057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:32.043057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.043057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:32.043057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:32.043057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:52:32.043057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:32.043057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:32.043057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.043057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:32.043311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:32.043311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:32.043311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.043311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:32.043311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:32.043311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:32.043311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:32.043311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:32.043311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:32.043311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:32.043311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:32.043311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:32.043311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:32.043311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:32.043629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:32.043629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:32.043629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.043629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:32.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:32.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:32.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:32.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:32.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:32.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:32.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:32.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:32.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:32.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:32.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:32.044036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:32.044505 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:32.044916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:32.044916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:32.044916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:32.044916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:32.044916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:32.044916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.044916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:32.059333 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:32.059566 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:32.059686 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:32.059823 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:32.060106 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:32.060422 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.060911 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:32.060911 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:32.060911 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:32.060911 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:32.060911 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:32.060911 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:32.060911 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.060911 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:32.061392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.061392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:32.061392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:32.061392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:32.061392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.061392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:32.061392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:32.061392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:32.061392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.061392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:32.061392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:32.061392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:32.061392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:32.061392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:32.061392 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:32.061704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:32.061704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:32.061704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:32.061704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:32.061704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:32.061704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:32.061704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:32.061704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.061704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:32.062017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.062017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:32.062017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:32.062017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:32.062017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.062017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:32.062017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:32.062017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:32.062017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:32.062017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.062017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:32.062233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:32.062233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:32.062233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.062233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:32.062233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:32.062233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:32.062233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:32.062233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:32.062233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:32.062233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:32.062233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:32.062233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:32.062233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:32.062233 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:32.062467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:32.062467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:32.062467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.062467 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.062711 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:32.063031 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:32.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:32.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:32.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:32.295627 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:32.295937 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:32.296067 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:32.296227 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:32.296560 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:32.296868 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:32.297029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.297640 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:32.297640 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:32.297640 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:32.297640 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:32.297640 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:32.297640 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:32.297640 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.297640 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:32.298309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.298309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:32.298309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:32.298309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:32.298309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.298309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:32.298309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:32.298309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.298309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:32.298309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:32.298309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:32.298309 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.298727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:32.298727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:32.298727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:32.298727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:32.298727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:32.298727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:32.298727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:32.298727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:32.298727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:32.298727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:32.298727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:32.298727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:32.298727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:32.299195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.299195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:32.299195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.299195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:32.299195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:32.299195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:32.299195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:32.299195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:32.299195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.299195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:32.299195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:32.299195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.299195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:32.299195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:32.299195 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:32.299732 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:32.300330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:32.300330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:32.300330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:32.300330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:32.300330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.300330 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:32.305834 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:32.305977 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:32.306109 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:32.306830 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:32.331850 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:32.331993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.332417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:32.332417 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:32.332417 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:32.332417 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:32.332417 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:32.332417 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:32.332417 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.332417 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:32.333024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.333024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:32.333024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:32.333024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:32.333024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.333024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:32.333024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:32.333024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.333024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:32.333024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:32.333024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:32.333024 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.333380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:32.333380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:32.333380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:32.333380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:32.333380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:32.333380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:32.333380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:32.333380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:32.333380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:32.333380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:32.333380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:32.333380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:32.333380 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:32.333726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.333726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:32.333726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.333726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:32.333726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:32.333726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:32.333726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:32.333726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:32.333726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.333726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:32.333726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:32.333726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.333726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:32.333726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:32.333726 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:32.334186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:32.334807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:32.334807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:32.334807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:32.334807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:32.334807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:32.334807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:36.985067 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:36.985607 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:36.985683 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:36.986410 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:36.986642 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:36.987238 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:36.987238 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:36.987238 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:36.987238 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:36.987238 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:36.987238 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:36.987238 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:36.987238 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:36.988149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:36.988149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:36.988149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:36.988149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:36.988149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:36.988149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:36.988149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:36.988149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:36.988149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:36.988149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:36.988149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:36.988149 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:36.988508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:36.988508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:36.988508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:36.988508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:36.988508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:36.988508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:36.988508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:36.988508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:36.988508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:36.988508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:36.988508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:36.988508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:36.988508 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:36.988761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:36.988761 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:36.989191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:36.989191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:36.989191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:36.989191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:36.989191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:36.989191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:36.989191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:36.989191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:36.989191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:36.989191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:36.989191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:36.989191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:36.989191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:36.989191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:36.989191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:36.989191 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:36.989445 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:36.989755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:36.989755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:36.989755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:37.040685 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:37.041059 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:37.041175 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:37.041277 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:52:37.041752 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:37.041932 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.042351 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:37.042351 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:37.042351 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:37.042351 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:37.042351 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:37.042351 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:37.042351 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.042351 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:37.042978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.042978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:37.042978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:37.042978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:37.042978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.042978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:37.042978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:37.042978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:37.042978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.042978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:37.042978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:37.042978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:37.042978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:37.042978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:37.042978 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:37.043221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:37.043221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:37.043221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:37.043221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:37.043221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:37.043221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:37.043221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:37.043221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.043221 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:37.043591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.043591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:37.043591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:37.043591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:37.043591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.043591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:37.043591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:37.043591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:52:37.043591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:37.043591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:37.043591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.043591 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:37.043832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:37.043832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:37.043832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.043832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:37.043832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:37.043832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:37.043832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:37.043832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:37.043832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:37.043832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:37.043832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:37.043832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:37.043832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:37.043832 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:37.044051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:37.044051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:37.044051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.044051 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:37.044303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.044303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:37.044303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:37.044303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:37.044303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:37.044303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:37.044303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.044303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:37.044303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:37.044303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.044303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:37.044303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:37.044303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.044303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:37.044303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:37.044303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:37.044875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:37.045442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:37.045442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:37.045442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:37.045442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:37.045442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:37.045442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.045442 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:37.059740 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:37.060012 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:37.060107 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:37.060207 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:37.061092 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:37.061092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.061215 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:37.061215 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:37.061215 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:37.061215 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:37.061215 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:37.061215 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:37.061215 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.061215 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:37.061755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.061755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:37.061755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:37.061755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:37.061755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.061755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:37.061755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:37.061755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:37.061755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.061755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:37.061755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:37.061755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:37.061755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:37.061755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:37.061755 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:37.061994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:37.061994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:37.061994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:37.061994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:37.061994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:37.061994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:37.061994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:37.061994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.061994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:37.062340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.062340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:37.062340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:37.062340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:37.062340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.062340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:37.062340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:37.062340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:37.062340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:37.062340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.062340 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:37.062552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:37.062552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:37.062552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.062552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:37.062552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:37.062552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:37.062552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:37.062552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:37.062552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:37.062552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:37.062552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:37.062552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:37.062552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:37.062552 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:37.062801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:37.062801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:37.062801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.062801 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.063122 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:37.063927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:37.064451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:37.064451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:37.064451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.064451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:37.296110 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:37.296559 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:37.296714 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:37.296878 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:37.297321 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:37.297524 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:37.297629 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.298120 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:37.298120 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:37.298120 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:37.298120 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:37.298120 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:37.298120 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:37.298120 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.298120 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:37.298838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.298838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:37.298838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:37.298838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:37.298838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.298838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:37.298838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:37.298838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.298838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:37.298838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:37.298838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:37.298838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.299050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:37.299050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:37.299050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:37.299050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:37.299050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:37.299050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:37.299050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:37.299050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:37.299050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:37.299050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:37.299050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:37.299050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:37.299050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:37.299261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.299261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:37.299533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.299533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:37.299533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:37.299533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:37.299533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:37.299533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:37.299533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.299533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:37.299533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:37.299533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.299533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:37.299533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:37.299533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.299533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:37.299533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:37.299533 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:37.299821 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:37.300134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:37.300134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.300134 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:37.306175 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:37.306409 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:37.306499 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:37.307128 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:37.332579 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:37.332754 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.333328 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:37.333328 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:37.333328 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:37.333328 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:37.333328 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:37.333328 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:37.333328 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.333328 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:37.334045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.027 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.334045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:37.334045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:37.334045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:37.334045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.334045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:37.334045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:37.334045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.334045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:37.334045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:37.334045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:37.334045 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.334269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:37.334269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:37.334269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:37.334269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:37.334269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:37.334269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:37.334269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:37.334269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:37.334269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:37.334269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:37.334269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:37.334269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:37.334269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:37.334481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.334481 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:37.334752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.334752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:37.334752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:37.334752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:37.334752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:37.334752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:37.334752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.334752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:37.334752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:37.334752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.334752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:37.334752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:37.334752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.334752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:37.334752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:37.334752 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:37.335057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:37.335327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:37.335327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:37.335327 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:41.985635 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:41.986113 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:41.986113 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:41.986447 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:41.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:41.987048 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:41.987048 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:41.987048 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:41.987048 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:41.987048 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:41.987048 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:41.987048 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:41.987048 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:41.987725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:41.987725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:41.987725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:41.987725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:41.987725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:41.987725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:41.987725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:41.987725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:41.987725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:41.987725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:41.987725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:41.987725 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:41.988015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:41.988015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:41.988015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:41.988015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:41.988015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:41.988015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:41.988015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:41.988015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:41.988015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:41.988015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:41.988015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:41.988015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:41.988015 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:41.988262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:41.988262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:41.988394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:41.988394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:41.988394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:41.988394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:41.988394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:41.988394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:41.988394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:41.988394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:41.988394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:41.988394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:41.988394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:41.988394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:41.988394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:41.988394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:41.988394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:41.988394 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:41.988727 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:41.989074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:41.989074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:41.989074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:42.041129 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:42.041487 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:42.041627 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:42.041743 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:52:42.042137 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:42.042332 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.042880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:42.042880 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:42.042880 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:42.042880 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:42.042880 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:42.042880 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:42.042880 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.042880 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:42.043413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.043413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:42.043413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:42.043413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:42.043413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.043413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:42.043413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:42.043413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:42.043413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.043413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:42.043413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:42.043413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:42.043413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:42.043413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:42.043413 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:42.043721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:42.043721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:42.043721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:42.043721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:42.043721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:42.043721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:42.043721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:42.043721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.043721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:42.044017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.044017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:42.044017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:42.044017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:42.044017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.044017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:42.044017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:42.044017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:52:42.044017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:42.044017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:42.044017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.044017 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:42.044224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:42.044224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:42.044224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.044224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:42.044224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:42.044224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:42.044224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:42.044224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:42.044224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:42.044224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:42.044224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:42.044224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:42.044224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:42.044224 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:42.044466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:42.044466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:42.044466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.044466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:42.044705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.044705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:42.044705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:42.044705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:42.044705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:42.044705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:42.044705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.044705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:42.044705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:42.044705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.044705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:42.044705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:42.044705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.044705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:42.044705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:42.044705 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:42.045013 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:42.045293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:42.045293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:42.045293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:42.045293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:42.045293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:42.045293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.045293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:42.060194 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:42.060443 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:42.060529 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:42.060671 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:42.060952 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:42.061137 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.061492 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:42.061492 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:42.061492 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:42.061492 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:42.061492 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:42.061492 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:42.061492 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.061492 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:42.062029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.062029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:42.062029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:42.062029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:42.062029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.062029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:42.062029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:42.062029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:42.062029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.062029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:42.062029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:42.062029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:42.062029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:42.062029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:42.062029 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:42.062293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:42.062293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:42.062293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:42.062293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:42.062293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:42.062293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:42.062293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:42.062293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.062293 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:42.062616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.062616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:42.062616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:42.062616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:42.062616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.062616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:42.062616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:42.062616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:42.062616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:42.062616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.062616 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:42.062838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:42.062838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:42.062838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.062838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:42.062838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:42.062838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:42.062838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:42.062838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:42.062838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:42.062838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:42.062838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:42.062838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:42.062838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:42.062838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:42.063113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:42.063113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:42.063113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.063113 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.063331 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:42.063674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:42.063988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:42.063988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:42.063988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.063988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:42.296664 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:42.296931 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:42.297031 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:42.297190 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:42.297476 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:42.297690 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:42.297782 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.298185 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:42.298185 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:42.298185 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:42.298185 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:42.298185 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:42.298185 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:42.298185 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.298185 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:42.298840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.298840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:42.298840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:42.298840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:42.298840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.298840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:42.298840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:42.298840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.298840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:42.298840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:42.298840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:42.298840 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.299086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:42.299086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:42.299086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:42.299086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:42.299086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:42.299086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:42.299086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:42.299086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:42.299086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:42.299086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:42.299086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:42.299086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:42.299086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:42.299313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.299313 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:42.299562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.299562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:42.299562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:42.299562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:42.299562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:42.299562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:42.299562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.299562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:42.299562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:42.299562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.299562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:42.299562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:42.299562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.299562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:42.299562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:42.299562 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:42.299896 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:42.300243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:42.300243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.300243 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:42.306501 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:42.306723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:42.306822 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:42.307417 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:42.332326 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:42.332511 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.332921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:42.332921 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:42.332921 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:42.332921 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:42.332921 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:42.332921 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:42.332921 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.332921 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:42.333561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.333561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:42.333561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:42.333561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:42.333561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.333561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:42.333561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:42.333561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.333561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:42.333561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:42.333561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:42.333561 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.334006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:42.334006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:42.334006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:42.334006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:42.334006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:42.334006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:42.334006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:42.334006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:42.334006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:42.334006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:42.334006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:42.334006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:42.334006 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:42.334482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.334482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:42.334482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.334482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:42.334482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:42.334482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:42.334482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:42.334482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:42.334482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.334482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:42.334482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:42.334482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.334482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:42.334482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:42.334482 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:42.334974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:42.335560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:42.335560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:42.335560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:42.335560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:42.335560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:42.335560 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:46.986056 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:46.986492 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:46.986613 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:46.987216 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:46.987433 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:46.988002 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:46.988002 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:46.988002 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:46.988002 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:46.988002 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:46.988002 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:46.988002 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:46.988002 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:46.988824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:46.988824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:46.988824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:46.988824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:46.988824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:46.988824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:46.988824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:46.988824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:46.988824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:46.988824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:46.988824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:46.988824 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:46.989063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:46.989063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:46.989063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:46.989063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:46.989063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:46.989063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:46.989063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:46.989063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:46.989063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:46.989063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:46.989063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:46.989063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:46.989063 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:46.989262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:46.989262 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:46.989729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:46.989729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:46.989729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:46.989729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:46.989729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:46.989729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:46.989729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:46.989729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:46.989729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:46.989729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:46.989729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:46.989729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:46.989729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:46.989729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:46.989729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:46.989729 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:46.990004 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:46.990280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:46.990280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:46.990280 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:47.041577 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:47.042079 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:47.042190 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:47.042312 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:52:47.042656 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:47.042834 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.043252 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:47.043252 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:47.043252 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:47.043252 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:47.043252 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:47.043252 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:47.043252 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.043252 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:47.043809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.043809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:47.043809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:47.043809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:47.043809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.043809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:47.043809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:47.043809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:47.043809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.043809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:47.043809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:47.043809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:47.043809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:47.043809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:47.043809 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:47.044107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:47.044107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:47.044107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:47.044107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:47.044107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:47.044107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:47.044107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:47.044107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.044107 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:47.044409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.044409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:47.044409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:47.044409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:47.044409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.044409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:47.044409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:47.044409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:52:47.044409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:47.044409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:47.044409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.044409 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:47.044929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:47.044929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:47.044929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.044929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:47.044929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:47.044929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:47.044929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:47.044929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:47.044929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:47.044929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:47.044929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:47.044929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:47.044929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:47.044929 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:47.045438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:47.045438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:47.045438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.045438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:47.045438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.045438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:47.045438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:47.045438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:47.045438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:47.045438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:47.045438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.045438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:47.045438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:47.045438 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:47.045747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:47.060561 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:47.060803 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:47.060908 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:47.061030 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:47.061330 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:47.061550 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.061975 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:47.061975 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:47.061975 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:47.061975 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:47.061975 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:47.061975 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:47.061975 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.061975 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:47.062483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.062483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:47.062483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:47.062483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:47.062483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.062483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:47.062483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:47.062483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:47.062483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.062483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:47.062483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:47.062483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:47.062483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:47.062483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:47.062483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:47.062780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:47.062780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:47.062780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:47.062780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:47.062780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:47.062780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:47.062780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:47.062780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.062780 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:47.063138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.063138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:47.063138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:47.063138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:47.063138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.063138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:47.063138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:47.063138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:47.063138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:47.063138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.063138 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:47.063344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:47.063344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:47.063344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.063344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:47.063344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:47.063344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:47.063344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:47.063344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:47.063344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:47.063344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:47.063344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:47.063344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:47.063344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:47.063344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:47.063598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:47.063598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:47.063598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.063598 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.063924 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:47.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:47.065207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:47.065207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:47.065207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.065207 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:47.297116 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:47.297373 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:47.297504 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:47.297727 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:47.298058 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:47.298306 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:47.298425 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.298927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:47.298927 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:47.298927 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:47.298927 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:47.298927 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:47.298927 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:47.298927 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.298927 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:47.299594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.299594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:47.299594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:47.299594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:47.299594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.299594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:47.299594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:47.299594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.299594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:47.299594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:47.299594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:47.299594 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.299811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:47.299811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:47.299811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:47.299811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:47.299811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:47.299811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:47.299811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:47.299811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:47.299811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:47.299811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:47.299811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:47.299811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:47.299811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:47.300019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.300019 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:47.300285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.300285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:47.300285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:47.300285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:47.300285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:47.300285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:47.300285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.300285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:47.300285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:47.300285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.300285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:47.300285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:47.300285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.300285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:47.300285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:47.300285 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:47.300575 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:47.300875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:47.300875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.300875 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:47.306827 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:47.307046 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:47.307131 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:47.307796 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:47.332408 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:47.332597 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.333011 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:47.333011 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:47.333011 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:47.333011 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:47.333011 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:47.333011 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:47.333011 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.333011 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:47.333630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.333630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:47.333630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:47.333630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:47.333630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.333630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:47.333630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:47.333630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.333630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:47.333630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:47.333630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:47.333630 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.333844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:47.333844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:47.333844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:47.333844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:47.333844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:47.333844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:47.333844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:47.333844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:47.333844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:47.333844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:47.333844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:47.333844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:47.333844 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:47.334050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.334050 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:47.334308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.334308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:47.334308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:47.334308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:47.334308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:47.334308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:47.334308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.334308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:47.334308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:47.334308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.334308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:47.334308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:47.334308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.334308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:47.334308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:47.334308 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:47.334617 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:47.334961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:47.334961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:47.334961 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:51.986525 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:51.987064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:51.987064 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:51.988198 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:51.988198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:51.988344 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:51.988344 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:51.988344 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:51.988344 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:51.988344 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:51.988344 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:51.988344 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:51.988344 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:51.989198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:51.989198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:51.989198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:51.989198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:51.989198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:51.989198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:51.989198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:51.989198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:51.989198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:51.989198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:51.989198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:51.989198 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:51.989475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:51.989475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:51.989475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:51.989475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:51.989475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:51.989475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:51.989475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:51.989475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:51.989475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:51.989475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:51.989475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:51.989475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:51.989475 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:51.989751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:51.989751 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:51.989969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:51.989969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:51.989969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:51.989969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:51.989969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:51.989969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:51.989969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:51.989969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:51.989969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:51.989969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:51.989969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:51.989969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:51.989969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:51.989969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:51.989969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:51.989969 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:51.990593 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:51.990880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:51.990880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:51.990880 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:52.042111 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:52.042549 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:52.042722 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:52.042891 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:52:52.043918 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:52.043918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.044105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:52.044105 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:52.044105 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:52.044105 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:52.044105 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:52.044105 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:52.044105 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.044105 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:52.044985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.044985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:52.044985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:52.044985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:52.044985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.044985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:52.044985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:52.044985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:52.044985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.044985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:52.044985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:52.044985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:52.044985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:52.044985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:52.044985 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:52.045250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:52.045250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:52.045250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:52.045250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:52.045250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:52.045250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:52.045250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:52.045250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.045250 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:52.045846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.045846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:52.045846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:52.045846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:52.045846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.045846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:52.045846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:52.045846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:52:52.045846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:52.045846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:52.045846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.045846 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:52.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:52.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:52.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:52.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:52.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:52.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:52.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:52.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:52.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:52.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:52.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:52.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:52.046057 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:52.046289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:52.046289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:52.046289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.046289 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:52.046884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.046884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:52.046884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:52.046884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:52.046884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:52.046884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:52.046884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.046884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:52.046884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:52.046884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.046884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:52.046884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:52.046884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.046884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:52.046884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:52.046884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:52.047164 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:52.047434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:52.047434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:52.047434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:52.047434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:52.047434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:52.047434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.047434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:52.061130 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:52.061596 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:52.061781 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:52.061957 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:52.062495 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:52.063341 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.063558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:52.063558 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:52.063558 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:52.063558 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:52.063558 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:52.063558 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:52.063558 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.063558 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:52.064326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.064326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:52.064326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:52.064326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:52.064326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.064326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:52.064326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:52.064326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:52.064326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.064326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:52.064326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:52.064326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:52.064326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:52.064326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:52.064326 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:52.064744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:52.064744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:52.064744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:52.064744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:52.064744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:52.064744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:52.064744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:52.064744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.064744 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:52.065371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.065371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:52.065371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:52.065371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:52.065371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.065371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:52.065371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:52.065371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:52.065371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:52.065371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.065371 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:52.065974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:52.065974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:52.065974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.065974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:52.065974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:52.065974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:52.065974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:52.065974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:52.065974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:52.065974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:52.065974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:52.065974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:52.065974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:52.065974 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:52.066807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:52.066807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:52.066807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.066807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:52.066807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.066807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:52.066807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:52.066807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:52.066807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:52.066807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:52.066807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.066807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:52.066807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:52.066807 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:52.067022 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:52.067369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:52.067369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:52.067369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:52.067369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:52.067369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:52.067369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:52.067369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:52.067369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:52.067369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:52.067369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.067369 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:52.297576 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:52.297940 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:52.298086 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:52.298223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:52.299141 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:52.299141 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:52.299141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.299290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:52.299290 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:52.299290 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:52.299290 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:52.299290 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:52.299290 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:52.299290 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.299290 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:52.299838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.299838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:52.299838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:52.299838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:52.299838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.299838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:52.299838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:52.299838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.299838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:52.299838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:52.299838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:52.299838 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.300074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:52.300074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:52.300074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:52.300074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:52.300074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:52.300074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:52.300074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:52.300074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:52.300074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:52.300074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:52.300074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:52.300074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:52.300074 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:52.300268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.300268 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:52.300469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.300469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:52.300469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:52.300469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:52.300469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:52.300469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:52.300469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.300469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:52.300469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:52.300469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.300469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:52.300469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:52.300469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.300469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:52.300469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:52.300469 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:52.300820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:52.301092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:52.301092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.301092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:52.307264 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:52.307626 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:52.307759 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:52.308530 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:52.336605 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:52.336605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.337033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:52.337033 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:52.337033 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:52.337033 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:52.337033 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:52.337033 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:52.337033 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.337033 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:52.337665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.337665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:52.337665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:52.337665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:52.337665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.337665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:52.337665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:52.337665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.337665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:52.337665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:52.337665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:52.337665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.337909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:52.337909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:52.337909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:52.337909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:52.337909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:52.337909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:52.337909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:52.337909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:52.337909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:52.337909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:52.337909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:52.337909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:52.337909 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:52.338125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.338125 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:52.338410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.338410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:52.338410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:52.338410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:52.338410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:52.338410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:52.338410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.338410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:52.338410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:52.338410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.338410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:52.338410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:52.338410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.338410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:52.338410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:52.338410 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:52.338760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:52.339094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:52.339094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:52.339094 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:56.986988 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:56.987432 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:56.987432 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:56.987912 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:56.988092 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:56.988515 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:56.988515 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:56.988515 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:56.988515 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:56.988515 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:56.988515 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:56.988515 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:56.988515 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:56.989255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:56.989255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:56.989255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:56.989255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:56.989255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:56.989255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:56.989255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:56.989255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:56.989255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:56.989255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:56.989255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:56.989255 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:56.989506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:56.989506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:56.989506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:56.989506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:56.989506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:56.989506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:56.989506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:56.989506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:56.989506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:56.989506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:56.989506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:56.989506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:56.989506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:56.989721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:56.989721 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:56.990001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:56.990001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:56.990001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:56.990001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:56.990001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:56.990001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:56.990001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:56.990001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:56.990001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:56.990001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:56.990001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:56.990001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:56.990001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:56.990001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:56.990001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:56.990001 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:56.990826 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:56.991247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:56.991247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:56.991247 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:57.042684 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:57.043013 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:57.043110 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:57.043212 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:52:57.043605 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:57.043772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.044218 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:57.044218 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:57.044218 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:57.044218 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:57.044218 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:57.044218 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:57.044218 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.044218 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:57.044836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.044836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:57.044836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:57.044836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:57.044836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.044836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:57.044836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:57.044836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:57.044836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.044836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:57.044836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:57.044836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:57.044836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:57.044836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:57.044836 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:57.045105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:57.045105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:57.045105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:57.045105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:57.045105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:57.045105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:57.045105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:57.045105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.045105 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:57.045525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.045525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:57.045525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:57.045525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:57.045525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.045525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:57.045525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:57.045525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:52:57.045525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:57.045525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:57.045525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.045525 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:57.046037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:57.046037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:57.046037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.046037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:57.046037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:57.046037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:57.046037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:57.046037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:57.046037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:57.046037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:57.046037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:57.046037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:57.046037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:57.046037 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:57.046261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:57.046261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:57.046261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.046261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:57.046261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.046261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:57.046261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:57.046261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:57.046261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:57.046261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:57.046261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.046261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:57.046261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:57.046261 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:57.046478 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:57.047345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:57.047345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:57.047345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:57.047345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:57.047345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:57.047345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:57.047345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:57.047345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:57.047345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:57.047345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:57.047345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.047345 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:57.061554 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:57.061830 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:57.061938 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:57.062056 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:57.062340 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:57.062520 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.062954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:57.062954 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:57.062954 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:57.062954 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:57.062954 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:57.062954 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:57.062954 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.062954 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:57.063495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.063495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:52:57.063495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:57.063495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:57.063495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.063495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:57.063495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:57.063495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:52:57.063495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.063495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:52:57.063495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:57.063495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:52:57.063495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:57.063495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:52:57.063495 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:52:57.063769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:57.063769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:52:57.063769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:57.063769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:52:57.063769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:52:57.063769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:57.063769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:52:57.063769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.063769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:52:57.064109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.064109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:57.064109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:57.064109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:57.064109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.064109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:57.064109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:57.064109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:57.064109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:57.064109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.064109 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:57.064307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:57.064307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:57.064307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.064307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:57.064307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:57.064307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:57.064307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:57.064307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:57.064307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:57.064307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:57.064307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:57.064307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:57.064307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:57.064307 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:57.064549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:57.064549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:57.064549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.064549 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.064841 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:57.065665 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:57.066071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:57.066071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:57.066071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.066071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:57.297984 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:57.298232 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:57.298298 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:57.298450 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:57.298740 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:57.298946 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:57.299039 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.299426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:57.299426 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:57.299426 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:57.299426 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:57.299426 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:57.299426 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:57.299426 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.299426 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:57.300014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.300014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:57.300014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:57.300014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:57.300014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.300014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:57.300014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:57.300014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.300014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:57.300014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:57.300014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:57.300014 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.300226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:57.300226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:57.300226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:57.300226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:57.300226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:57.300226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:57.300226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:57.300226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:57.300226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:57.300226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:57.300226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:57.300226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:57.300226 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:57.300426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.300426 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:57.300710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.300710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:57.300710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:57.300710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:57.300710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:57.300710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:57.300710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.300710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:57.300710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:57.300710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.300710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:57.300710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:57.300710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.300710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:57.300710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:57.300710 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:57.300988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:57.301259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:57.301259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.301259 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:52:57.307498 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:52:57.307669 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:52:57.307760 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:52:57.308290 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:52:57.331617 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:52:57.331854 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.332186 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:52:57.332186 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:57.332186 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:52:57.332186 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:52:57.332186 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:57.332186 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:52:57.332186 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.332186 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:52:57.332808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.024 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.332808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:52:57.332808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:57.332808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:52:57.332808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.332808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:57.332808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:57.332808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.332808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:57.332808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:57.332808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:52:57.332808 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.333040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:52:57.333040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:57.333040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:52:57.333040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:57.333040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:52:57.333040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:52:57.333040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:57.333040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:52:57.333040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:57.333040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:52:57.333040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:52:57.333040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:57.333040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:52:57.333227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.333227 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:52:57.333484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.333484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:52:57.333484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:52:57.333484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:52:57.333484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:52:57.333484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:52:57.333484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.333484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:52:57.333484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:52:57.333484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.333484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:52:57.333484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:52:57.333484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.333484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:52:57.333484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:52:57.333484 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:52:57.333814 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:52:57.334193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:52:57.334193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:52:57.334193 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:01.987325 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:01.987920 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:01.988031 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:01.988451 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:01.988605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:01.989036 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:01.989036 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:01.989036 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:01.989036 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:01.989036 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:01.989036 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:01.989036 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:01.989036 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:01.989659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:01.989659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:01.989659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:01.989659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:01.989659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:01.989659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:53:01.989659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:01.989659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:01.989659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:01.989659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:01.989659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:01.989659 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:01.989988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:01.989988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:01.989988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:01.989988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:01.989988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:01.989988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:01.989988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:01.989988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:01.989988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:01.989988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:01.989988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:01.989988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:01.989988 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:01.990301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:01.990301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:01.990301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:01.990301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:01.990301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:01.990301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:01.990301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:01.990301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:01.990301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:01.990301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:01.990301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:01.990301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:01.990301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:01.990301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:01.990301 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:01.990704 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:01.991162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:01.991162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:01.991162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:01.991162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:01.991162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:01.991162 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:02.043094 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:02.043466 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:02.043583 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:02.043694 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:53:02.044038 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:02.044217 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.044658 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:02.044658 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:02.044658 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:02.044658 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:02.044658 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:02.044658 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:02.044658 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.044658 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:02.045203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.045203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:53:02.045203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:02.045203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:02.045203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.045203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:53:02.045203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:02.045203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:53:02.045203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.045203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:53:02.045203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:02.045203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:53:02.045203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:02.045203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:53:02.045203 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:53:02.045459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:02.045459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:53:02.045459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:02.045459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:53:02.045459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:53:02.045459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:02.045459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:53:02.045459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.045459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:53:02.045990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.045990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:02.045990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:02.045990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:02.045990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.045990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:53:02.045990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:53:02.045990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:53:02.045990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:02.045990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:02.045990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.045990 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:02.046417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:02.046417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:02.046417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.046417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:02.046417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:02.046417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:02.046417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:02.046417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:02.046417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:02.046417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:02.046417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:02.046417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:02.046417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:02.046417 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:02.046668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:02.046668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:02.046668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.046668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:02.046668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.046668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:02.046668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:02.046668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:02.046668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:02.046668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:02.046668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.046668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:02.046668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:02.046668 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:02.046899 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:02.047213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:02.047213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:02.047213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:02.047213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:02.047213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:02.047213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:02.047213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:02.047213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:02.047213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:02.047213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:02.047213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.047213 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:02.061973 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:02.062223 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:02.062314 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:02.062405 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:53:02.062704 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:02.062897 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.063269 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:02.063269 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:02.063269 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:02.063269 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:02.063269 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:02.063269 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:02.063269 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.063269 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:02.063845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.063845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:53:02.063845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:02.063845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:02.063845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.063845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:53:02.063845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:02.063845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:53:02.063845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.063845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:53:02.063845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:02.063845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:53:02.063845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:02.063845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:53:02.063845 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:53:02.064493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:02.064493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:53:02.064493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:02.064493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:53:02.064493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:53:02.064493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:02.064493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:53:02.064493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.064493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:53:02.064493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.064493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:02.064493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:02.064493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:02.064493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.064750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:53:02.064750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:53:02.064750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:02.064750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:02.064750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.064750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:02.064750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:02.064750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:02.064750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.064750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:02.064750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:02.064750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:02.064750 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:02.064957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:02.064957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:02.064957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:02.064957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:02.064957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:02.064957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:02.064957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:02.064957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:02.064957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:02.064957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.064957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.065458 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:02.066412 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:02.066746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:02.066746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:02.066746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.066746 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:02.298814 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:02.299317 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:02.299442 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:02.299700 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:53:02.300105 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:02.300319 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:02.300424 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.300965 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:02.300965 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:02.300965 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:02.300965 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:02.300965 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:02.300965 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:02.300965 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.300965 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:02.301999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.301999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:02.301999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:02.301999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:02.301999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.301999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:53:02.301999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:02.301999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.301999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:02.301999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:02.301999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:02.301999 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.302222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:02.302222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:02.302222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:02.302222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:02.302222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:02.302222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:02.302222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:02.302222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:02.302222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:02.302222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:02.302222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:02.302222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:02.302222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:02.302429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.302429 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:02.302918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.302918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:02.302918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:02.302918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:02.302918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:02.302918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:02.302918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.302918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:02.302918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:02.302918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.302918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:02.302918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:02.302918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.302918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:02.302918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:02.302918 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:02.303174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:02.303470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:02.303470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.303470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:02.307778 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:02.308105 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:02.308279 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:02.309125 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:53:02.336491 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:02.336772 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.337354 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:02.337354 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:02.337354 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:02.337354 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:02.337354 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:02.337354 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:02.337354 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.337354 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:02.338142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.029 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.338142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:02.338142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:02.338142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:02.338142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.338142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:53:02.338142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:02.338142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.338142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:02.338142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:02.338142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:02.338142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.338389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:02.338389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:02.338389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:02.338389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:02.338389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:02.338389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:02.338389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:02.338389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:02.338389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:02.338389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:02.338389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:02.338389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:02.338389 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:02.338634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.338634 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:02.338935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.338935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:02.338935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:02.338935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:02.338935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:02.338935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:02.338935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.338935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:02.338935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:02.338935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.338935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:02.338935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:02.338935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.338935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:02.338935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:02.338935 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:02.339769 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:02.340303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:02.340303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:02.340303 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:06.988109 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:06.988635 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:06.988635 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:06.989723 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:06.989723 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:06.989902 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:06.989902 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:06.989902 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:06.989902 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:06.989902 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:06.989902 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:06.989902 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:06.989902 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:06.990673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:06.990673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:06.990673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:06.990673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:06.990673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:06.990673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:53:06.990673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:06.990673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:06.990673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:06.990673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:06.990673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:06.990673 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:06.990937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:06.990937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:06.990937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:06.990937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:06.990937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:06.990937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:06.990937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:06.990937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:06.990937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:06.990937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:06.990937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:06.990937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:06.990937 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:06.991144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:06.991144 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:06.991491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:06.991491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:06.991491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:06.991491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:06.991491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:06.991491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:06.991491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:06.991491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:06.991491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:06.991491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:06.991491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:06.991491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:06.991491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:06.991491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:06.991491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:06.991491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:06.992311 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:06.992877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:06.992877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:06.992877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:07.043553 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:07.043958 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:07.044130 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:07.044265 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:53:07.044638 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:07.044820 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.045286 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:07.045286 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:07.045286 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:07.045286 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:07.045286 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:07.045286 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:07.045286 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.045286 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:07.045987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.045987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:53:07.045987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:07.045987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:07.045987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.045987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:53:07.045987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:07.045987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:53:07.045987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.045987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:53:07.045987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:07.045987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:53:07.045987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:07.045987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:53:07.045987 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:53:07.046459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:07.046459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:53:07.046459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:07.046459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:53:07.046459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:53:07.046459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:07.046459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:53:07.046459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.046459 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:53:07.046747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.046747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:07.046747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:07.046747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:07.046747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.046747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:53:07.046747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:53:07.046747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:53:07.046747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:07.046747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:07.046747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.046747 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:07.047102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:07.047102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:07.047102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.047102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:07.047102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:07.047102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:07.047102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:07.047102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:07.047102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:07.047102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:07.047102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:07.047102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:07.047102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:07.047102 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:07.047470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:07.047470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:07.047470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.047470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:07.047470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.047470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:07.047470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:07.047470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:07.047470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:07.047470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:07.047470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.047470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:07.047470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:07.047470 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:07.047938 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:07.048434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:07.048434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:07.048434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:07.048434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:07.048434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:07.048434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:07.048434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:07.048434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:07.048434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:07.048434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:07.048434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.048434 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:07.062373 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:07.062785 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:07.062939 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:07.063061 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:53:07.063498 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:07.063811 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.064395 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:07.064395 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:07.064395 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:07.064395 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:07.064395 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:07.064395 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:07.064395 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.064395 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:07.065155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.065155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:53:07.065155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:07.065155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:07.065155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.065155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:53:07.065155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:07.065155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:53:07.065155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.065155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:53:07.065155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:07.065155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:53:07.065155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:07.065155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:53:07.065155 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:53:07.065632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:07.065632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:53:07.065632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:07.065632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:53:07.065632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:53:07.065632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:07.065632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:53:07.065632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.065632 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:53:07.065901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.065901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:07.065901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:07.065901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:07.065901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.065901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:53:07.065901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:53:07.065901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:07.065901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:07.065901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.065901 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:07.066362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:07.066362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:07.066362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.066362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:07.066362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:07.066362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:07.066362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:07.066362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:07.066362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:07.066362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:07.066362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:07.066362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:07.066362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:07.066362 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:07.066753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:07.066753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:07.066753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.066753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:07.066753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.066753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:07.066753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:07.066753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:07.066753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:07.066753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:07.066753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.066753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:07.066753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:07.066753 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:07.067174 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:07.067709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:07.067709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:07.067709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:07.067709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:07.067709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:07.067709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:07.067709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:07.067709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:07.067709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:07.067709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.067709 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:07.299335 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:07.299501 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:07.299572 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:07.299733 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:53:07.300166 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:07.300360 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:07.300493 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.300900 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:07.300900 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:07.300900 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:07.300900 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:07.300900 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:07.300900 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:07.300900 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.300900 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:07.301557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.301557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:07.301557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:07.301557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:07.301557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.301557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:53:07.301557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:07.301557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.301557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:07.301557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:07.301557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:07.301557 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.301984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:07.301984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:07.301984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:07.301984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:07.301984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:07.301984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:07.301984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:07.301984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:07.301984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:07.301984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:07.301984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:07.301984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:07.301984 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:07.302382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.302382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:07.302382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.302382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:07.302382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:07.302382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:07.302382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:07.302382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:07.302382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.302382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:07.302382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:07.302382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.302382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:07.302382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:07.302382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:07.302894 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:07.303428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:07.303428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:07.303428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:07.303428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:07.303428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.303428 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:07.308136 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:07.308346 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:07.308463 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:07.309278 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:53:07.333310 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:07.333491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.334040 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:07.334040 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:07.334040 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:07.334040 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:07.334040 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:07.334040 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:07.334040 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.334040 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:07.334858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.026 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.334858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:07.334858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:07.334858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:07.334858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.334858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:53:07.334858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:07.334858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.334858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:07.334858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:07.334858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:07.334858 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.335175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:07.335175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:07.335175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:07.335175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:07.335175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:07.335175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:07.335175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:07.335175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:07.335175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:07.335175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:07.335175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:07.335175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:07.335175 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:07.335400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.335400 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:07.335545 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.335545 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:07.335545 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:07.335545 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:07.335545 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:07.335545 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:07.335545 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.335545 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:07.335545 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:07.335545 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.335545 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:07.335545 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:07.335545 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.335545 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:07.335545 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:07.335545 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:07.335873 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:07.336254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:07.336254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:07.336254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:11.988583 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:11.989143 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:11.989143 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:11.990558 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:11.990558 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:11.990764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:11.990764 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:11.990764 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:11.990764 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:11.990764 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:11.990764 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:11.990764 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:11.990764 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:11.991565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:11.991565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:11.991565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:11.991565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:11.991565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:11.991565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:53:11.991565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:11.991565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:11.991565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:11.991565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:11.991565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:11.991565 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:11.992501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:11.992501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:11.992501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:11.992501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:11.992501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:11.992501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:11.992501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:11.992501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:11.992501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:11.992501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:11.992501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:11.992501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:11.992501 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:11.993180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:11.993180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:11.993180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:11.993180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:11.993180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:11.993180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:11.993180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:11.993180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:11.993180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:11.993180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:11.993180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:11.993180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:11.993180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:11.993180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:11.993180 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:11.993652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:11.994189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:11.994189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:11.994189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:11.994189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:11.994189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:11.994189 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:12.044003 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:12.044413 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:12.044564 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:12.044697 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:53:12.045994 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:12.045994 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.046142 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:12.046142 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:12.046142 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:12.046142 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:12.046142 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:12.046142 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:12.046142 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.046142 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:12.046962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.046962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:53:12.046962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:12.046962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:12.046962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.046962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:53:12.046962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:12.046962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:53:12.046962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.046962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:53:12.046962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:12.046962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:53:12.046962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:12.046962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:53:12.046962 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:53:12.047638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:12.047638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:53:12.047638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:12.047638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:53:12.047638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:53:12.047638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:12.047638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:53:12.047638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.047638 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:53:12.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:12.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:12.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:12.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:53:12.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:53:12.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:53:12.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:12.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:12.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.047884 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:12.048161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:12.048161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:12.048161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.048161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:12.048161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:12.048161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:12.048161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:12.048161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:12.048161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:12.048161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:12.048161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:12.048161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:12.048161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:12.048161 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:12.048491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:12.048491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:12.048491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.048491 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:12.048608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.048608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:12.048608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:12.048608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:12.048608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:12.048608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:12.048608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.048608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:12.048608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:12.048608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.048608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:12.048608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:12.048608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.048608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:12.048608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:12.048608 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:12.048993 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:12.049406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:12.049406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:12.049406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:12.049406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:12.049406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:12.049406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.049406 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:12.062743 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:12.063186 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:12.063332 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:12.063445 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:53:12.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:12.064554 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.064712 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:12.064712 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:12.064712 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:12.064712 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:12.064712 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:12.064712 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:12.064712 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.064712 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:12.065514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.065514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:53:12.065514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:12.065514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:12.065514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.065514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:53:12.065514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:12.065514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:53:12.065514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.065514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:53:12.065514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:12.065514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:53:12.065514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:12.065514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:53:12.065514 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:53:12.065921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:12.065921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:53:12.065921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:12.065921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:53:12.065921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:53:12.065921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:12.065921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:53:12.065921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.065921 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:53:12.066466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.066466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:12.066466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:12.066466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:12.066466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.066466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:53:12.066466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:53:12.066466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:12.066466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:12.066466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.066466 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:12.067032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:12.067032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:12.067032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.067032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:12.067032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:12.067032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:12.067032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:12.067032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:12.067032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:12.067032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:12.067032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:12.067032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:12.067032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:12.067032 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:12.067254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:12.067254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:12.067254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.067254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:12.067254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.067254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:12.067254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:12.067254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:12.067254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:12.067254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:12.067254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.067254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:12.067254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:12.067254 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:12.067506 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:12.068033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:12.068033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:12.068033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:12.068033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:12.068033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:12.068033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:12.068033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:12.068033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:12.068033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:12.068033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.068033 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:12.299639 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:12.299990 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:12.300097 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:12.300258 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:53:12.301222 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:12.301222 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:12.301222 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.301314 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:12.301314 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:12.301314 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:12.301314 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:12.301314 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:12.301314 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:12.301314 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.301314 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:12.301889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.301889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:12.301889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:12.301889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:12.301889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.301889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:53:12.301889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:12.301889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.301889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:12.301889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:12.301889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:12.301889 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.302494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:12.302494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:12.302494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:12.302494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:12.302494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:12.302494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:12.302494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:12.302494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:12.302494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:12.302494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:12.302494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:12.302494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:12.302494 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:12.303248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.303248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:12.303248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.303248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:12.303248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:12.303248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:12.303248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:12.303248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:12.303248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.303248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:12.303248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:12.303248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.303248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:12.303248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:12.303248 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:12.303605 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:12.303927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:12.303927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:12.303927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:12.303927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:12.303927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.303927 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:12.308432 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:12.308674 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:12.308756 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:12.309350 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:53:12.332337 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:12.332337 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.332451 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:12.332451 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:12.332451 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:12.332451 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:12.332451 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:12.332451 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:12.332451 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.332451 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:12.333071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.024 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.333071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:12.333071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:12.333071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:12.333071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.333071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:53:12.333071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:12.333071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.333071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:12.333071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:12.333071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:12.333071 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.333284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:12.333284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:12.333284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:12.333284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:12.333284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:12.333284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:12.333284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:12.333284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:12.333284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:12.333284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:12.333284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:12.333284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:12.333284 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:12.333483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.333483 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:12.333829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.333829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:12.333829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:12.333829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:12.333829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:12.333829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:12.333829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.333829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:12.333829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:12.333829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.333829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:12.333829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:12.333829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.333829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:12.333829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:12.333829 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:12.334479 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:12.334786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:12.334786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:12.334786 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:16.988957 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:16.989455 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_router_default_route_empty_dst_ip {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:16.989455 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Remove default routes with empty destination from routers. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:16.990463 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DelStaticRoutesCommand(_result=None, lrouter=neutron-946b4709-7598-480b-a9d5-cb9e971cea74, routes=[], if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:16.990463 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:16.990641 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:16.990641 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:16.990641 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:16.990641 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:16.990641 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:16.990641 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:16.990641 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:16.990641 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:16.991366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_router_default_route_empty_dst_ip failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:16.991366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:16.991366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:16.991366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:16.991366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:16.991366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:53:16.991366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:16.991366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:16.991366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:16.991366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:16.991366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:16.991366 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:16.991686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:16.991686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:16.991686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:16.991686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:16.991686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:16.991686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:16.991686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:16.991686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:16.991686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:16.991686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:16.991686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:16.991686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:16.991686 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:16.991954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:16.991954 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:16.992169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_router_default_route_empty_dst_ip' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:16.992169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:16.992169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:16.992169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:16.992169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:16.992169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:16.992169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:16.992169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:16.992169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:16.992169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:16.992169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:16.992169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:16.992169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:16.992169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:16.992169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:16.992169 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 984, in check_router_default_route_empty_dst_ip Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:16.992976 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:16.993671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:16.993671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:16.993671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:17.044471 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:17.044810 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: set_fip_distributed_flag {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:17.044949 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Set distributed floating IP flag in NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:17.045059 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting fip-distributed flag in NB_Global to False {{(pid=89427) set_fip_distributed_flag /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:1208}} Jun 01 11:53:17.045968 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('external_ids', {'neutron:fip-distributed': 'False'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:17.045968 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.046114 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:17.046114 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:17.046114 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:17.046114 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:17.046114 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:17.046114 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:17.046114 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.046114 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:17.046877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.046877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:53:17.046877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:17.046877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:17.046877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.046877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:53:17.046877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:17.046877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:53:17.046877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.046877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:53:17.046877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:17.046877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:53:17.046877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:17.046877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:53:17.046877 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:53:17.047141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:17.047141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:53:17.047141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:17.047141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:53:17.047141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:53:17.047141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:17.047141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:53:17.047141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.047141 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:53:17.047518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task set_fip_distributed_flag failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.047518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:17.047518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:17.047518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:17.047518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.047518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:53:17.047518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:53:17.047518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^ Jun 01 11:53:17.047518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:17.047518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:17.047518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.047518 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:17.048007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:17.048007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:17.048007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.048007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:17.048007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:17.048007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:17.048007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:17.048007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:17.048007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:17.048007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:17.048007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:17.048007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:17.048007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:17.048007 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:17.048457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:17.048457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:17.048457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.048457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:17.048457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.set_fip_distributed_flag' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.048457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:17.048457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:17.048457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:17.048457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:17.048457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:17.048457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.048457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:17.048457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:17.048457 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1212, in set_fip_distributed_flag Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ovn_const.OVN_FIP_DISTRIBUTED_KEY: str(distributed)}).execute( Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^ Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:17.048916 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:17.049202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:17.049202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:17.049202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:17.049202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:17.049202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:17.049202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:17.049202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:17.049202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:17.049202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:17.049202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:17.049202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.049202 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:17.063159 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:17.063491 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_ha_failover {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:17.063621 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply HA chassis failover BFD settings to NB_Global. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:17.063736 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:53:17.064623 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'bfd-min-rx': '1000', 'bfd-min-tx': '100', 'bfd-mult': '3'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:17.064623 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.064764 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:17.064764 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:17.064764 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:17.064764 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:17.064764 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:17.064764 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:17.064764 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.064764 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:17.065382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Error executing command (DbSetCommand): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.065382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): Jun 01 11:53:17.065382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:17.065382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:17.065382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.065382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:53:17.065382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:17.065382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with super().transaction(*args, **kwargs) as t: Jun 01 11:53:17.065382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.065382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) Jun 01 11:53:17.065382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:17.065382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command with self.create_transaction( Jun 01 11:53:17.065382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:17.065382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() Jun 01 11:53:17.065382 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^ Jun 01 11:53:17.065651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:17.065651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise result.ex Jun 01 11:53:17.065651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:17.065651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) Jun 01 11:53:17.065651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command ^^^^^^^^^^^^^^^ Jun 01 11:53:17.065651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:17.065651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) Jun 01 11:53:17.065651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.065651 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.command Jun 01 11:53:17.066086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_ha_failover failed after 0.002 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.066086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:17.066086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:17.066086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:17.066086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.066086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:53:17.066086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:53:17.066086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:17.066086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:17.066086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.066086 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:17.066290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:17.066290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:17.066290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.066290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:17.066290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:17.066290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:17.066290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:17.066290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:17.066290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:17.066290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:17.066290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:17.066290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:17.066290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:17.066290 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:17.066530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:17.066530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:17.066530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.066530 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_ha_failover' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.066966 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 1273, in update_ha_failover Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self._nb_idl.set_nb_global_options(**options).execute( Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 49, in execute Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.api.transaction(check_error, log_errors, **kwargs) as t: Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:17.067288 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:17.067581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:17.067581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:17.067581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.067581 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:17.300035 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:17.300296 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: update_mac_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:17.300418 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Update MAC binding aging and router MAC age limits. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:17.300623 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'mac_binding_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:53:17.300932 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'mac_binding_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:17.301671 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): SetLRouterMacAgeLimitCommand(_result=None, router=None, threshold=300) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:17.301671 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.301785 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:17.301785 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:17.301785 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:17.301785 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:17.301785 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:17.301785 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:17.301785 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.301785 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:17.302628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task update_mac_aging_settings failed after 0.001 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.302628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:17.302628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:17.302628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:17.302628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.302628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:53:17.302628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:17.302628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.302628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:17.302628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:17.302628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:17.302628 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.302898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:17.302898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:17.302898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:17.302898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:17.302898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:17.302898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:17.302898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:17.302898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:17.302898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:17.302898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:17.302898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:17.302898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:17.302898 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:17.303096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.303096 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:17.303652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.update_mac_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.303652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:17.303652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:17.303652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:17.303652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:17.303652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:17.303652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.303652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:17.303652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:17.303652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.303652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:17.303652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:17.303652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.303652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:17.303652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:17.303652 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 900, in update_mac_aging_settings Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:17.303930 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:17.304232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:17.304232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.304232 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Jun 01 11:53:17.308682 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG futurist.periodics [-] Submitting periodic callback 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' {{(pid=89427) _process_scheduled /opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py:784}} Jun 01 11:53:17.308880 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Starting OVN maintenance task: check_fdb_aging_settings {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:99}} Jun 01 11:53:17.308975 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task: Apply FDB aging limits to NB_Global and provider switches. {{(pid=89427) wrapper /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py:101}} Jun 01 11:53:17.309649 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Setting NB_Global options: {'fdb_removal_limit': '0'} {{(pid=89427) set_nb_global_options /opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py:951}} Jun 01 11:53:17.333674 np0000001841 neutron-ovn-maintenance-worker[89427]: DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(_result=None, table=NB_Global, record=., col_values=dict_items([('options', {'fdb_removal_limit': '0'})]), if_exists=True) {{(pid=89427) do_commit /opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}} Jun 01 11:53:17.333674 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [-] OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.333853 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR ovsdbapp.backend.ovs_idl.transaction [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Traceback (most recent call last): Jun 01 11:53:17.333853 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:17.333853 np0000001841 neutron-ovn-maintenance-worker[89427]: txn.results.put(txn.do_commit()) Jun 01 11:53:17.333853 np0000001841 neutron-ovn-maintenance-worker[89427]: ^^^^^^^^^^^^^^^ Jun 01 11:53:17.333853 np0000001841 neutron-ovn-maintenance-worker[89427]: File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:17.333853 np0000001841 neutron-ovn-maintenance-worker[89427]: raise RuntimeError(msg) Jun 01 11:53:17.333853 np0000001841 neutron-ovn-maintenance-worker[89427]: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.333853 np0000001841 neutron-ovn-maintenance-worker[89427]: Jun 01 11:53:17.334588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] OVN maintenance task check_fdb_aging_settings failed after 0.025 seconds: RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.334588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last): Jun 01 11:53:17.334588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:17.334588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance result = f(*args, **kwargs) Jun 01 11:53:17.334588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.334588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:53:17.334588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:17.334588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.334588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:17.334588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:17.334588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with super().transaction(*args, **kwargs) as t: Jun 01 11:53:17.334588 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.334957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen) Jun 01 11:53:17.334957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:17.334957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance with self.create_transaction( Jun 01 11:53:17.334957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:17.334957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit() Jun 01 11:53:17.334957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^ Jun 01 11:53:17.334957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:17.334957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex Jun 01 11:53:17.334957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:17.334957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit()) Jun 01 11:53:17.334957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ^^^^^^^^^^^^^^^ Jun 01 11:53:17.334957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:17.334957 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RuntimeError(msg) Jun 01 11:53:17.335283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.335283 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Jun 01 11:53:17.335760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics [None req-41439c5c-446d-4a27-ba2f-8cd3b2ea7635 None None] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_fdb_aging_settings' (it runs every 5.00 seconds): RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.335760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics Traceback (most recent call last): Jun 01 11:53:17.335760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 376, in run Jun 01 11:53:17.335760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics work() Jun 01 11:53:17.335760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 90, in __call__ Jun 01 11:53:17.335760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return self.callback(*self.args, **self.kwargs) Jun 01 11:53:17.335760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.335760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/futurist/periodics.py", line 213, in decorator Jun 01 11:53:17.335760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(*args, **kwargs) Jun 01 11:53:17.335760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.335760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 81, in decorator Jun 01 11:53:17.335760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics return f(self, *args, **kwargs) Jun 01 11:53:17.335760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.335760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 107, in wrapper Jun 01 11:53:17.335760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics result = f(*args, **kwargs) Jun 01 11:53:17.335760 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^^^^ Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 884, in check_fdb_aging_settings Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self._nb_idl.transaction(check_error=True) as txn: Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/neutron/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 272, in transaction Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with super().transaction(*args, **kwargs) as t: Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics next(self.gen) Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 114, in transaction Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics with self.create_transaction( Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/api.py", line 71, in __exit__ Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics self.result = self.commit() Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^ Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 64, in commit Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise result.ex Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 127, in run Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics txn.results.put(txn.do_commit()) Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics ^^^^^^^^^^^^^^^ Jun 01 11:53:17.336339 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics File "/opt/stack/data/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 123, in do_commit Jun 01 11:53:17.336695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics raise RuntimeError(msg) Jun 01 11:53:17.336695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics RuntimeError: OVSDB Error: The transaction failed because the IDL has been configured to require a database lock but didn't get it yet or has already lost it Jun 01 11:53:17.336695 np0000001841 neutron-ovn-maintenance-worker[89427]: ERROR futurist.periodics